Video ingest and analysis pipeline
The pipeline behind every video upload, from presigned multipart transfer to indexed scenes, built so nothing heavy touches the request path.
A video upload fans out into parallel background work that produces preview assets, video derivatives, transcription, and indexed scene intelligence.
The system separates fast media work (metadata, thumbnails, scrubs) from long-running analysis (transcoding, transcription, detection) by routing them to different queues backed by different worker images. What leaves the pipeline is a structured, searchable asset, not just a stored file.
The engineering problem was keeping those branches observable, failure-isolated, and independently switchable per organization, without any of it touching the upload request or blocking the file from appearing in the library.
The browser asks the API for presigned part URLs, then uploads 10MB parts directly to object storage. The API sees the request that opens the upload and the request that closes it, never the file itself. Nothing about the size of a video affects request handling.
Three parts are in flight at any time. Each part retries up to three times with exponential backoff, carries its own abort signal, and is timed out independently. A single flaky part costs one part, not the upload.
Once every part has an ETag, one finalize call completes the multipart upload, records the original as a tracked artifact, and enqueues the background branches. It returns before any of them start, and a duplicate finalize is a no-op rather than a second pipeline.
Finalizing an upload enqueues independent background tasks, each routed to a queue sized for its runtime. Three run for every video. Two are per-organization. None of them block the upload response, and none of them can starve another.
Every background task is routed by name to one of a set of queues grouped into runtime bands: instant, fast, medium, and heavy. Queues map to worker images and to Kubernetes deployments, so a queue is also a scaling unit and a dependency boundary. A worker that only dispatches jobs doesn't ship a media toolchain.
The ingest path alone crosses six of them. Thumbnails and sprites go to the media queue. The analysis orchestrator and the watermark derivative go to transcode-class queues. Chunk dispatch, the long-running detection work, and the callback-driven subtitle work are three separate detection queues. Outbound indexing sits on its own queue so a slow external call can't occupy a worker that something latency-sensitive needs.
The routing table is the honest description of the system. Reading it tells you what runs where and what competes with what, without reading a single task body.
Ten candidate frames are sampled between 20% and 80% of the duration, deliberately skipping the weak openings and endings common in uploaded footage. Each is scored on sharpness, contrast, and saturation, normalized against the best candidate in the set, and the strongest frame wins. The result ships as AVIF, falling back to WebP if the encode fails.
Frames sampled on a fixed interval are tiled into a contact sheet and a hover-preview sprite sheet, so the library can scrub a timeline without loading the video. Videos too short to fill a sheet skip it rather than producing a degenerate one.
Duration, dimensions, frame rate, and codec details are probed on the first download and written back immediately. That makes the asset usable in the library within seconds, long before any analysis completes.
The orchestrator runs an integrity and duration probe before it spawns anything. A corrupted file is marked terminally and the branch ends there. No transcription request, no detection jobs, no partial job state to reconcile later.
Transcription is dispatched immediately after the integrity check, not after the transcode. It only needs an extracted audio track, so making it wait on a full video transcode would add minutes to the longest path in the pipeline for no reason. It is the first branch out and, with shot detection, one of the two the join waits on.
A normalized copy is produced next, and the stages after it read that copy instead of the raw upload. Chunking, detection, and per-scene cutting all get consistent codec, frame rate, and resolution assumptions, which removes an entire category of format edge cases from the downstream code.
Which branches exist at all is read from per-organization flags at the moment the tasks are scheduled. One flag opens the full analysis fan-out. A narrower one runs transcription for captions only and stops before scene assembly. A third routes to a compliance-detection path instead. The workers themselves stay unconditional.
Transcription branches off ahead of the transcode. Speech timing, chunked face analysis, and full-video shot detection branch off after it. Only transcription and shot detection gate scene assembly. Face analysis matters later, when the asset's final status is decided.
Each branch creates a durable job row keyed by file and job type, carrying status, a completion fraction, the external job id it is waiting on, structured metadata, and its last error. Active work is queryable in SQL. Completion updates the row instead of vanishing into the task runner.
Face detection splits the transcoded copy into time-based chunks and dispatches one job per chunk. Each chunk carries its own row with its time window, its storage location, and its result location. The parent's completion fraction is recomputed from its children on every callback.
A chunk callback recounts its siblings rather than trusting a counter. The parent only resolves when nothing is outstanding, and it resolves to failed if any chunk failed. Aggregation fires once, guarded against a second callback re-running it, and only after the parent row has been committed.
If a video has no audio track, no transcription request is sent. The pipeline instead writes evenly spaced placeholder scene chunks and marks the transcription job complete. Scene assembly then runs on exactly the same code path, with no special case for silent video.
Assembly aligns transcript-derived scene boundaries to shot boundaries, fills any remaining gaps so the full duration is covered, cuts each scene into its own clip, and runs per-scene structured extraction with the neighbouring scenes as context. The result is written as scene rows and indexed for search.
The transcript is segmented at its largest silence gaps, which produces semantically coherent scene candidates from speech alone. Those candidates exist before shot detection returns, which is exactly why the two have to be reconciled rather than concatenated.
Speech does not cover a whole video. Assembly extends short trailing gaps into the preceding scene, promotes medium ones into a scene of their own, and splits long ones into evenly sized scenes. Coverage of the full duration is a property of the output, not a hope.
The transcription callback also enqueues subtitle generation on a separate queue. Caption artifacts are produced and persisted on their own schedule, and a failure there never reaches the scene-assembly join.
Retrieval and ranking are a separate system, built by other people. What the pipeline owes them is a document per scene that is correct, replaceable, and still correct after the library changes underneath it. That contract is the last thing this pipeline is responsible for.
Each scene is written as a standalone document carrying its own copy of the video-level fields it needs, plus its workspace id. The extracted signal (people, objects, locations, actions, emotions, brands, on-screen text, synopsis, mood, dialogue) sits flat alongside the transcript and shot boundaries. Tenancy is a filter on the document, not a join at query time.
Per-scene extraction returns event times relative to the scene's own clip. Those get rebased to absolute video time, then pulled onto a shot boundary whenever they land within 200ms of one. Without that step a result seeks to a frame or two before the cut it's describing, which reads as a bug in search rather than in the pipeline that fed it.
Indexing a video first removes every existing document for it, then bulk-writes the new set in small batches. Re-running analysis replaces the video's scenes instead of duplicating them, so a retry is safe. The same call also normalizes the extracted entities into relational tables, so the structured view and the searchable view are written together or not at all.
Because every document carries its own copy of the video's title and location, the index drifts the moment the library changes. Renaming a video rewrites its documents in place, copying one into a project reindexes them under the new id, and deleting one clears them by query. That maintenance is the price of not joining at read time, and it belongs to the pipeline, not to search.
One upload becomes a dozen objects in storage: the original, a thumbnail, a scrub sheet, a sprite sheet, a normalized transcode, a watermarked copy, a streaming package, extracted audio, analysis chunks, detection results, transcripts. Left untracked, that is an unbounded storage bill and a deletion path nobody can prove correct.
Each one is instead recorded as a row naming its bucket, its key, its type, its size, and two things that make cleanup decidable: whether it survives with the file or is disposable, and whether it can be regenerated from something else. The original is retained and not regenerable. A sprite sheet is regenerable. A chunk is deleted after processing.
Rows are unique on file, type, and a hash of the storage location, so a re-run overwrites its entry rather than accumulating duplicates of work it already did.
Status changes publish to a realtime channel named for the workspace and folder the asset lives in. Clients subscribe to the view they have open, so a busy workspace doesn't push updates to people looking at something else.
A new file invalidates the folder query and refetches. A progress tick does not. It writes directly into the cached list, updating only the page holding that asset and preserving object identity everywhere else.
Progress events arrive continuously while a video indexes. Refetching on each one, or cloning the cache to update it, made every tick cost the whole loaded list and re-render every card. Patching in place turns a per-tick full render into one row.
Transcoding, transcription, and detection cannot sit on the upload request. The finalize call returns before any background task starts, and every expensive step is behind a queue.
Face detection dispatches many independent chunk jobs per video. Without deriving parent state from the children on every callback, completion is ambiguous, and one chunk failing has to fail the parent without silently discarding the rest.
Transcription and shot detection run independently and finish in either order, so whichever lands second starts assembly. If one never reaches a terminal state, assembly never starts, which is why every branch has to end somewhere explicit rather than just stopping.
Parallel branches all report into one percentage, and they race. Today each writer clamps its own contribution to a ceiling, which keeps the number sane in practice but is a convention rather than an invariant. A monotonic write would make it one.
A newer orchestrator runs alongside the task-chained flow this pipeline is built on, and some branches check which one they're under before firing. That check is load-bearing, and it's the first thing that gets stale.
Grouping tasks by how long they run rather than which product they belong to means a two-hour transcode and a two-second thumbnail never share a worker pool.
Integrity validation happens before any downstream job is created. A bad file fails cheaply instead of spawning orphaned work across several services.
Chunked analysis derives parent state from its children on every callback. The parent advances only when nothing is outstanding, and aggregation runs exactly once.
Silent video still produces placeholder scene structure, so assembly runs on the same code path with no special case.
Which branches exist is decided when tasks are enqueued. Workers stay simple and the active pipeline is readable from the scheduling call, at the cost of a flag flip not affecting work already in flight.
A metadata check that skipped already-conformant files exists but is switched off. Every video gets normalized, which costs compute on files that didn't need it and buys one set of assumptions downstream.
Transcription and shot detection must both complete before assembly starts. If one is slow, assembly waits. There is no partial scene output.
Newer and older approaches overlap. The current upload path is clear, but the codebase still shows its evolutionary history, and some processors sit in it without being wired into the default flow.
Thumbnail, scrub, and technical metadata land seconds after upload. The library shows a real asset, with live progress on it, while the long branches are still running.
Assembly produces structured, indexed scenes covering the full duration of every video that completes the analysis path, supporting search, tagging, comments, and repurpose workflows.
Flags let organizations sit on different combinations: full analysis, captions only, compliance detection, streaming. New branches ship without touching the upload path or the organizations that don't want them.
Splitting queues by how long work takes is not overengineering. Without it, one slow transcode delays thumbnail generation for every other upload in the system, and the symptom looks like the thumbnail code being slow.
A corruption check that ends the branch on a bad file is the highest-leverage line in the orchestrator. The alternative is orphaned jobs across several services and no clean terminal state to reconcile them against.
Transcription needs an audio track, not a normalized video, so it leaves before the transcode. Auditing a pipeline for dependencies that are sequential by habit rather than by requirement is usually the cheapest latency win available.
A join on two independent results is only tractable because each has a durable terminal state, and because whichever finishes second is the one that starts the next stage. Without job rows, the join degenerates into polling against unstable state.
Multi-track timeline in the browser