Case: A File Upload Service
Upload, Storage, Metadata, Permissions, Download. The first version streams through the server; the requirement that changes the shape is size, and the prototype question — can the browser upload a large file directly to object storage? — is answered in an afternoon before any architecture depends on it.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You need to let users upload files, and someone says "just use object storage". What are the five capabilities, which one forces the architecture, and what is the smallest prototype that tells you whether the browser can talk to storage directly?
The store needs product images from admins, and a later requirement mentions "customers attach files to support tickets, sometimes large ones". You have written a form upload before — the file arrives at the server and you write it to disk — and you do not know whether that survives large files, many uploads, or a second server.
Keep the server in the middle. The browser posts the file, the server receives it, the server writes it somewhere — disk today, object storage tomorrow — and the download path reads it back. It is the pattern every framework documents and it works on the first try.
It works for images and stalls for the support-ticket case: a large file ties up a server worker for the whole upload, the server's memory or disk becomes the ceiling, and the second server cannot see the first one's disk.
- It works for images and stalls for the support-ticket case: a large file ties up a server worker for the whole upload, the server's memory or disk becomes the ceiling, and the second server cannot see the first one's disk.
- Permissions are an afterthought because the demo had one user. "Who can download this file?" arrives when the first customer attaches something private to a ticket, and the download endpoint has no idea who is asking.
- Metadata lives in the filename. Size, type, owner, the ticket it belongs to — all recoverable from the path until the path changes, and then not.
- The "just use object storage" advice is followed as a storage swap only — the server still proxies every byte — so the architecture keeps its bottleneck and gains a new dependency.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Decompose into the five capabilities the sentence hides: Upload (bytes get from the browser to somewhere), Storage (they persist), Metadata (what the file is and whose), Permissions (who may read it), Download (bytes get back). The first three are the skeleton; the last two are the product (Decomposition by Capability).
- Find the requirement that forces the shape. It is size, and its neighbour is concurrency: a server that proxies bytes is bounded by its workers and its disk. Ask the question that decides the architecture: can the browser send the file directly to object storage, with the server only granting permission and recording metadata? (A Prototype Answers a Question)
- Answer it with a prototype before anything depends on the answer — a page, a signed upload URL, a large file, and an observation of what actually happened. The prototype is the research; the architecture follows from its result (Prototype vs Production).
- Keep metadata and permissions in your database whatever the storage is. Object storage holds bytes; your system holds the fact that this file belongs to that ticket and may be read by these users. Download is then "check permission, then hand out a short-lived link" (Source of Truth).
Five capabilities, five homes
The decomposition separates what the reflex merges. Upload and Storage look like one thing until a large file shows they have different bottlenecks; Metadata and Permissions look like part of Storage until a private file shows they must live in your database. Each leaf has an observation that would show it works, and each can be built and tested without the ones after it.
- ├Upload— bytes from the browser to somewhere; the capability that size forces
- └Issue a signed upload URLtestable The server returns a URL usable once, for one key, for a short window; a second use or a different key is rejected by storage.
- └Complete and verifytestable After the browser reports completion, the server confirms the object exists with the expected size before marking the file READY.
- ├Storage— bytes persist, independent of any server
- └Object keyed by file idtestable Two uploads with the same user-supplied filename produce two objects; no key is derived from user input.
- ├Metadata— what the file is and whose — your database's job
- └File row with statetestable A file has owner, size, type, key and a state (UPLOADING → READY); the row exists before the first byte is sent.
- ├Permissions— who may read; answered by your rules, not by the bucket
- └One check for every downloadtestable The ticket's customer and support can fetch the attachment; another customer receives not-found; product images pass with no login.
- ├Download— bytes back, through the permission check
- └Short-lived signed read linktestable A download link works inside its window and fails after; the server never streams the bytes itself.
Product images use the same tree with a proxied Upload leaf and a public Download leaf. The decomposition is the same; the answers differ by requirement.
The question the prototype answered
The prototype was an experiment with a prediction, not a first draft of the feature. The board below is the state before it ran: what was known, what was assumed, and the question with its experiment. Notice that the second unknown was discovered by the first experiment — a prototype that answers one question usually asks another (Unknown Unknowns).
- ✓Product images: small, public, admin-uploaded; the proxied path is fine and stays.
- ✓Ticket attachments: private, sometimes large, customer-uploaded; the proxied path ties up a worker per upload.
- ✓Bytes will live in object storage; metadata and permissions will live in our database.
- ~One storage region and one provider in V1 — written down so the residency question is known to be waiting (When the Data Is Not Allowed to Leave in Distributed).
- ~A short-lived read link is acceptable to the support tool; to be checked, since a link that expires mid-session is a support complaint.
? Big files.
becomes Can the browser upload a large file directly to object storage using a URL our server signs, and what does our server do while that happens?
experiment One page, one signed-URL endpoint, one large test file; predict "works, server idle"; run it with the server's request log open.
? The size ceiling.
becomes Above what size does the provider require a multipart upload, and does the browser flow change or only the signing?
experiment Read the provider's limit; upload one file just below and one just above; note which step differs.
? Half-finished uploads.
becomes If the browser closes after the signed URL is issued, what exists in storage and in our database, and who removes it?
experiment Start an upload, kill the tab, list the bucket and the File rows; write the sweep rule from what is found.
The first unknown was the one that decided the architecture. The other two arrived from running it, and neither blocked anything.
Proxied or direct
The decision, with the criteria rather than a winner. The store uses both: proxied for product images, direct for attachments. The criterion is the forcing requirement — size and concurrency — and the cost column is what each path adds to the system.
Should bytes pass through our server or go from the browser to storage directly?
when Files are small, uploads are infrequent, and you want one code path with no signing and no verify step. Product images.
cost Every byte occupies a server worker; memory or disk is the ceiling; a second server cannot see the first one's temporary files unless you already write straight to storage.
when Files can be large or numerous, or the server must not be the bottleneck. Ticket attachments.
cost A signed-URL endpoint, a completion-and-verify step, a File state machine, an orphan sweeper, and a provider-specific multipart flow above the size ceiling (Choosing an Upload Path in Backend weighs the same choice).
when Files are very large or connections unreliable, and a failed upload must continue rather than restart.
cost Part bookkeeping on both sides and a more complex client; justified only when the observation shows uploads failing mid-way often enough to matter.
How to do it
Most important first.
- V1 for product images: browser → server → disk or object storage, with a File row (id, owner, size, type, storage key, created). Small files, one admin, no permission question yet. Ship it.
- When the large-file requirement appears, write the prototype question down precisely and run it: a signed URL, a large file, the browser, a stopwatch, and a look at what the server did in the meantime (Spikes).
- Decide the upload path from the result — direct-to-storage with the server issuing signed URLs, or server-proxied — and write the decision with the size at which it flips.
- Model permissions as a question the download endpoint asks your database, never the storage: "may this user read file X?" Then issue a short-lived download link (Where Authorization Must Live in Security covers the enforcement).
- Add the boring failure cases before real users: an upload that starts and never finishes, a file whose type lies, a storage call that fails after the metadata row was written (Partial Failure).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- The store's product images, V1. Admin picks an image, the browser posts it, the server writes it to object storage and a File row to the database, the product references the File. Download is a public URL because product images are public. It took a morning and it is the right amount of engineering for images.
- The prototype, when the ticket-attachment requirement arrived. Question: "can the browser upload a large file directly to object storage using a URL my server signs, and what does my server do while it happens?" Setup: one page, one endpoint that returns a signed upload URL for a key, one large test file. Prediction, written first: it will work and the server will be idle. Observation: it worked; the server handled one small request; the upload time depended only on the client's connection. A second observation nobody predicted: the storage provider imposes a per-request size ceiling above which the upload must be split into parts, which is a follow-up question, not a blocker (An Unknown Is Not a Blocker).
- The architecture that followed. Upload: server issues a signed URL and creates a File row in state UPLOADING; browser sends bytes to storage; browser tells the server it finished; server verifies the object exists and marks the row READY. Storage: object storage, keyed by File id, never by user-supplied name. Metadata: the File row. Permissions: a ticket attachment is readable by the ticket's customer and by support — a rule in the database. Download: check the rule, sign a short-lived read URL, redirect (Direct Uploads and Signed Authorization in Cloud covers the storage side).
- The failure cases, injected. Browser closes mid-upload → the File row stays UPLOADING; a sweep deletes rows that never became READY and their orphaned objects. Client claims "finished" but the object is missing → the server's verify step catches it; the row is not READY. Storage write succeeds, metadata write fails → the object is orphaned until the sweep finds it, and the user is told to retry; the order was chosen so that the failure leaves an orphan rather than a dangling row.
How you know it worked
What now exists that did not before, and what question you can now ask.
- The five capabilities each have a home, and you can say which system is authoritative for each — bytes in storage, everything else in your database.
- The direct-upload question was answered by an observation, with a prediction written before it, and the architecture cites it.
- Every download passes through one permission check that asks your database, and public product images and private attachments use the same path with different rules.
- A file in state UPLOADING has a defined fate, and orphaned objects have a sweeper.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Which of the five capabilities does this requirement actually change, and which can stay as they are?
- ?What requirement forces the architecture, and what prototype would show whether the simpler shape survives it?
- ?Which system is authoritative for the bytes, and which for who may read them?
- ?What is the state of a file whose upload never finished, and who cleans it up?
What can go wrong
- The direct-upload architecture is built for product images too, because it is "more scalable". Product images are small and public; the proxied path was correct for them, and the extra complexity bought nothing.
- The prototype becomes the implementation: the signed-URL endpoint from the spike, with no File row and no verify step, ships. It answered a question; it was never designed to hold state (Prototype vs Production).
- Permissions are delegated to storage — a bucket per customer, say — because it seems simpler. Your system then cannot answer "who may read this?" without asking the provider, and support's access is a second mechanism.
- The multipart ceiling discovered in the prototype is treated as a blocker and the whole direct-upload design is abandoned, when it was one more question with a documented answer.
- Direct-to-storage upload takes your server out of the byte path and puts a signed-URL flow, a verify step and an orphan sweeper in its place. For small files the proxied path is less code and easier to reason about.
- Keeping permissions in your database means every download is a database check plus a signed link — slower than a public URL, and correct for private files. The store uses both paths on purpose.
- Running the prototype cost an afternoon before the feature; skipping it would have cost the redesign when the first large attachment hit the proxied path in production.
- "Object storage is always the answer for files." It is the answer for bytes at any real volume. It is not the answer for metadata or permissions, and treating it as one is how systems lose track of who owns what.
- "Direct upload from the browser is insecure." A signed URL is scoped to one key, one method and a short window; the server still decides who gets one. The security question moves; it does not disappear (File Upload Security in Security).
- "The prototype proved the architecture." It proved that the browser can send bytes to storage with a signed URL. Multipart, resumption, virus scanning, and what happens at the provider's size ceiling were separate questions, and the case treated them that way.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALFive capabilities with separate homes, and a prototype for the forcing requirement, hold for any system that moves large blobs — video, backups, exports — though the forcing requirement is not always size; sometimes it is retention or residency.
- DOMAIN-SPECIFICFor an internal tool with small files and a trusted set of users, the server-proxied path with a File row is the whole design and direct-to-storage is unjustified complexity; for a product accepting large or numerous uploads from the public, the direct path is usually the first version. The case has both inside one store.
- ILLUSTRATIVEThe support-ticket requirement, the prototype's observations, the provider's size ceiling and the orphan sweeper are invented to show the shape of the argument; no provider limits or timings are being reported.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's delegation cards at /manifesto/delegating: the storage SDK signs the URL for you; whether the signed URL is scoped narrowly enough is a question you still own.