01
Related
Where to read next on the same subject.
02
Short answer
Moving a Spark estate to Dataproc is a sequence, not a lift. Inventory every job with its owner, runtime, and cost. Move the storage off the cluster first, so compute becomes disposable. Convert the jobs against a pinned runtime, then validate outputs, runtimes, and cost against a recorded baseline before the cutover window. The old cluster stays warm for exactly one cycle.
I have lost count of the migrations that began with someone spinning up the new cluster and "just moving the jobs over." The job list turns out to be three spreadsheets and a memory. Two of the jobs run nothing but their own history. One is load-bearing and undocumented. The new cluster arrives optimistically sized, and the first real week of runtimes looks like a mystery instead of a measurement.
The sequence below exists because that first week can be made boring, and boring is the goal.
03
Inventory the estate before touching anything
Every job gets a row: schedule, owner, runtime, library versions, data locations, and cost. The owner column is the hard one, because half the estate will trace back to someone who left, and several jobs will have the answer "nobody knows." That answer is a finding, not an embarrassment. A job with no owner and no documented purpose is a retirement candidate, and the cheapest migration is the one you do not perform.
The inventory also decides treatment. Jobs that only exist by habit get killed, in the open, with a notice period. Jobs that are trivial get a straight conversion. The load-bearing few get the full treatment: conversion, validation on sampled and full data, and a named watcher through the first cycle after cutover.
04
Move the storage first
This is the step that changes the character of everything after it. Data moves from HDFS to object storage before any job changes platforms. Once the data lives outside the cluster, compute becomes disposable: clusters can be resized, replaced, and thrown away without touching the data, which is the property a migration keeps needing.
Carry the formats and schemas with the data, and keep the source read-only until cutover. The read-only rule has saved me from the classic dual-write mystery more than once: two systems accepting changes, one of them quietly losing, and a reconciliation exercise that nobody budgeted.
05
Convert the jobs against a pinned runtime
The conversions cluster into a handful of real work. Runtime parity comes first: the Spark and Python versions on the new platform should match the old estate, or the differences should be chosen deliberately, with the release notes read rather than assumed. The Dataproc image release notes and the Spark release documentation for the versions involved are the reference here, and reading them is cheaper than debugging a behavior change in production.
Libraries get pinned into a requirements file, all of them, including the transitive dependencies that were implicit on the old cluster. Paths get rewritten for object storage, and connector behavior changes are handled at one seam, in one place, rather than scattered through job code. The tests that passed on the source platform run unchanged on the new one; a conversion that needs its tests rewritten has also, quietly, changed its behavior.
06
Validate against the baseline table
The baseline table is the migration's contract with reality: per job, the recorded runtime and cost on the old platform, measured before the move. Validation means running both jobs, old and new, on sampled data first and then full volumes, comparing outputs row by row rather than by eyeball, and comparing runtimes and cost against the table.
Deltas get explained, not absorbed. A new job that runs slower gets tuned, or the reason gets written down and accepted. A job that runs faster gets its number trusted once, verified, and then enjoyed. What cannot happen is the shrug, because the shrug at validation becomes the shrug at the month-end cost review, and that one has an audience.
07
Cutover in one controlled window
The schedule flips in a single window, with the old cluster kept warm for exactly one cycle. Monitoring and paging are live before the flip, not assembled after. The rollback path is written and rehearsed, which is why it will almost certainly not be needed.
The first cycle after cutover belongs to watching: runtimes against the table, rejects, downstream loads, and the cost trend. The retirement of the old cluster is a decision recorded with a date, not an act of faith at the cutover window.
08
What good looks like
The migration report reads like a table: every job, its baseline, its new runtime, its new cost, and its validation result. The jobs that were killed are listed with their retirement dates. The estate is smaller than it was, the compute is disposable, and the next platform change is an afternoon of repointing rather than a quarter of archaeology.
09
Sequencing with the people who run the jobs
The inventory has authors, and the authors are the fastest conversion tool available. The engineer who wrote the UDF knows why it exists; the analyst who owns the monthly job knows which downstream report breaks if it slips. Bring the job owners into the sequence at inventory time, and the conversion plan writes itself in their vocabulary: which jobs convert cleanly, which need a library hunt, and which have a dependency that only surfaces on the last day of the month.
The owners also hold the baselines. The recorded runtime table should be measured with them, on a normal day, because a baseline taken during the quarterly close misrepresents every other week. Their presence turns the validation conversation from a dispute into a comparison: the job that ran forty minutes on the old cluster runs fifty-two on the new one, and the owner can say immediately whether the twelve minutes matter to the schedule that consumes the output.
Twin-run windows do the rest. For the load-bearing jobs, run old and new in parallel for one cycle on the same inputs, with the outputs compared by the reconciliation tests. The owners read the diffs and classify them: a formatting change the report tolerates, a null handling difference that needs a fix, a genuine numeric delta that stops the line. That classification cannot be made from a distance, and it cannot be made after cutover.
The cutover window itself gets owner coverage too. Someone who understands each critical job watches its first scheduled run on the new platform, in real time, with the old cluster still warm. Most first runs pass and the watching feels excessive. The one that does not pass justifies every hour of it, because the difference between a five-minute fix and a missed load window is usually a person who recognized the failure mode on sight.
10
Related
The ingestion guide shows the event path these jobs feed. The reconciliation guide supplies the validation layer this sequence depends on. The logistics case study carries the CDC pattern on Google Cloud at production scale.