Migrating between systems

Shipped

A migration here is a shape, not a separate product: rows one connected system has already mirrored into your instance are read, reshaped, and written so that a second connected system creates them. A customer arriving with years of Xero history and a brand-new NetSuite account wants exactly that, and every piece it needs already ships — a process to declare the work, ctx.query to read the mirror, and the write boundary to create the result and push it upstream.

There is no bulk import endpoint and no migration screen. This page is the shape that works today, the three limits it runs into, and the records it cannot move at all — so you learn them here rather than against a customer’s data.

The shape

One process, one invocation per source row, and the record naming the system it belongs to:

import { defineProcess } from '@turbobase/extensions';

// The type parameter is what gives `ctx.input` its shape; the `input` schema
// below is what the invoke route validates the body against.
export default defineProcess<{ externalId: string }>({
  id: 'acme:migrate-customer',
  input: {
    type: 'object',
    properties: { externalId: { type: 'string' } },
    required: ['externalId'],
  },
  steps: [
    {
      id: 'customer',
      recordType: 'customer',
      build: async (ctx) => {
        // `ctx.query` takes a query STRING, so anything you interpolate into a
        // filter has to be encoded — an unescaped `&` or `,` in a value is read
        // as more query, and silently widens the read.
        const id = encodeURIComponent(ctx.input.externalId);
        const [row] = (await ctx.query(
          `entity?select=company_name,email,external_id&entity_type=eq.customer` +
            `&source_system=eq.xero&external_id=eq.${id}`,
        )) as { company_name: string; email: string | null; external_id: string }[];
        if (!row) throw new Error(`nothing mirrored for ${ctx.input.externalId}`);
        return {
          parent: {
            company_name: row.company_name,
            email: row.email,
            // The record names the system it belongs to.
            source_system: 'netsuite',
            raw: { migrated_from: { source_system: 'xero', external_id: row.external_id } },
          },
        };
      },
    },
  ],
});

Deploy it like any other extension, then ask for it once per row you are moving:

turbobase ext deploy processes/migrate-customer.ts

// POST /v1/processes/:id — 202 { runId, processId, state: 'queued' }
await fetch('/v1/processes/acme:migrate-customer', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ externalId: 'XERO-C-1' }),
});

The record names its target, not your session

On a create, parent.source_system names which connected system the new record belongs to, and it is the value the row is stamped with. That is what makes a migration possible at all from a session signed in against the system you are migrating away from: the write routes by the record’s own hint, never by the session’s claim. Invoked under a Xero session, the process above still creates a NetSuite record, and the write trail queues it for the NetSuite adapter.

Two things to know about the field:

• Naming a system that is not connected is refused — 422 with SOURCE_NOT_CONNECTED — rather than accepted and left with nowhere to sync.
• On an update it is verified, not written. A record already belongs to a system; restating it is fine, a different value is 422 SOURCE_MISMATCH — or, on a record type that means a different record in different systems (account is a customer in Zoho CRM and Salesforce and a ledger account everywhere else), 404 row not found or not permitted, because on those the field is also what chose the table the row was looked for in. Either way you cannot hand an existing record to another system by relabelling it. A migration creates; it never re-points.
• On a custom table it is verified, not written — on a create too. A table mirrored from one system, or authored on the instance, already says which system owns its rows, so a create addressed by its slug or by the customrecord_-prefixed spelling of it takes that answer and a disagreeing source_system is 422 SOURCE_MISMATCH. Those tables are per-install and are not what a migration moves; the record types above are.

On an instance with one system connected the field can be left off and defaults. Name it anyway in a migration — the whole point is that there are two.

One invocation per source row

A process declares its steps once and the list is fixed when the module is deployed — what varies is how many times the platform runs it. The plain shape below is one invocation per source row, driven by whatever is doing the migrating: your app under a signed-in session, a { kind: 'process' } schedule, or a script outside the browser — turbobase processes invoke, or a call with a personal access token carrying extensions:deploy whose owner is an administrator; the run writes as that owner. See Keeping a long run going. Two declarations change that, and both are the right answer more often than fanning out from a caller:

• eachRow makes the process a migration: one invocation runs the whole step list once per outstanding source row, a slice at a time, queues its own next slice, and each step reads its row from ctx.row. See A migration recipe for the create-then-mark-the-row shape, and Moving many records.
• forEach repeats a GROUP of steps once per element of an array on the invoked input — for a chain whose length the caller decides, up to the instance’s expansion cap. See Repeating a group of steps.

Three consequences worth planning for:

• Each run is its own unit. One failing invocation does not stop the others. Each carries its own ledger — GET /v1/processes/runs/:runId — with its own steps and its own refusals.
• Idempotency is per run for a chain, and per SOURCE ROW for a migration. A run redelivered to a worker resumes where it stopped and creates nothing twice. Re-invoking a plain chain with identical input mints a new run, and that run creates a second record — nothing knows the two invocations meant the same Xero customer, so guard it yourself: stamp provenance on the record, as the example above does, and look for it in build before returning a payload. A process that declares eachRow needs none of that: its write key is derived from the row’s own key, so re-offering a row is a no-op.
• It will not go faster than the source. Writes of one record type reach a source strictly one at a time, in order, and outbound calls per source are gated. Ten thousand invocations enqueued at once queue.

The three limits

What	Today	Constant
Rows one POST /v1/batch sub-query returns (not ctx.query — see below)	1,000	READ_MAX_ROWS
Bytes one ctx.query returns	12 MB	MAX_BATCH_RESPONSE_BYTES
How long a step waits for the source	5 minutes	DEFAULT_AWAIT_SOURCE_BUDGET_MS

The last column is there so this page can be held to it: a test fails the build if a number here stops matching the platform, rather than leaving you to find out from a run.

ctx.query hands back the whole result, or throws. Unlike a browser’s POST /v1/batch read, it has no row cap: a step reads in order to decide what to write, and a silently shortened answer would migrate the first thousand records and report success. What bounds it is the byte ceiling above — a read over it fails the step with the limit named. Read a page explicitly (limit and offset) when a page is what you want, or let eachRow take the slice for you, as the example above does.

awaitSource waits on a wall clock, and in bulk it will run out. Declare it on a step only when a later step in the same run needs the id the source minted for an earlier one — the ordering it buys is paid for by holding the run open. Behind a queue of other writes of the same record type, five minutes is not a generous budget; it is the ordinary case. A wait that runs out is not a failure: nothing was refused, the write is on the trail, and the run ends committed with the steps it never reached in outstanding and its reason naming the step still waiting. The record that step created is still there, still on its way to the source. For a bulk migration, leave awaitSource off: let each record go, and watch source health for what the source said.

What cannot be migrated at all

A step naming a record type that is read-only for the target system is refused at the write boundary before anything is created. The run ends failed, and the step’s ledger row carries WRITE_REFUSED with the message record type "subsidiary" is not writable — and no field, because the refusal is about the type rather than about anything in your payload.

subsidiary on NetSuite is the live case: it is mirrored inbound and has no write-back path, and neither do NetSuite’s other setup lists such as accountingPeriod and currency, so a step that tries to create one is refused. The chart of accounts is not among them: account on NetSuite is writable, so accounts can be migrated like any other record. The refusal arrives at the first step rather than half-way through a run, which is the right failure — but reading a record type’s direction in the data model first is cheaper than discovering it.

The same refusal covers a record type the target system does not have at all, not only one it holds read-only. Record types are not global: a message belongs to the systems that carry messages, and naming one in a step that targets a system without them is refused here with the same WRITE_REFUSED and the same sentence. Which record types a system has is what the data model lists per source.

Nothing migrates until the target’s write-back is on

Write-back ships off for every source, so a new instance is read-only towards the system you are migrating into. While it is off, the first step of the first run is refused at the write boundary before anything is created: the run ends failed, the step’s ledger row carries SYNC_BIDIRECTIONAL_DISABLED with a message telling you to contact us, and its pg_row_id is null. Turn write-back on for the target system before you migrate anything — this is the first thing a migration runs into, not the last.

The rollback reads the same entitlement. onFailure: 'rollback' deletes what the run created, newest first, through the delete boundary, and that boundary asks the same question the write asked. So if write-back is switched off between a run’s writes and its rollback, the undo cannot run: the run ends failed with a reason beginning rollback incomplete, and the records it created are still live.

A timed-out awaitSource step reaches that same rollback incomplete ending by a different route, and that one needs no flag change. Its record is still pending on the trail with no verdict, so the rollback deliberately leaves it rather than tombstone a record the source may be about to create.

Before you run one

• Turn write-back on for the target system. With it off nothing is created at all — the first step is refused.
• Read the direction of every record type your steps name. A read-only type is a wall, not a slow path.
• Decide your re-run story. Nothing keys idempotency on the source row.
• Leave awaitSource off unless a later step needs the id.
• Run it for a handful of rows first and read each run back — turbobase processes runs or GET /v1/processes/runs/:runId — before you turn the volume up. There is no offline dry-run for a process: ext test and ext replay cover hooks and conduits, and turbobase sandbox does not seed an instance yet. A few rows against the real target is the rehearsal that exists.

Next steps

• Writing processes — the full defineProcess contract.
• Writing conduits — bridge a record between sources as it arrives, rather than in bulk.
• Data model — which record types are bidirectional, and what write-back is.
• Sandboxes and testing — how extensions execute, and the offline dry-runs that exist for hooks and conduits.