Mantle Open the web app

Internals

The reference behind Architecture: how the parts work in detail, with the numbers and the workarounds that a change has to respect. Read Architecture first: each part there links to its paragraphs here. Each fact here belongs to code that names it, so a change to that code should change its paragraph.

Part What is here
The web build Starting the core in a browser tab: the lock, the worker, the files, the sealing, the save as a tab closes and its measurements.
What start does The lock on the data folder, and the order the core is built in.
Saving, loading and going back Failed saves, removing a profile, and reading a file a newer version wrote. With the costs, measured.
Parts of the core One note for each of the core's smaller parts: downloads, history, notes, friends, the service's limits, keys.
Profile PINs, Sharing a setup What a PIN does and does not keep, and what a share holds.
The household's plugins, A profile's own lists Records that sync, and the upgrade from the betas before 0.3.0.
Telling what friends and shared lists did What the rounds note of other people's doing, told once; device names; what deleting takes.
What sync costs the service The two kinds of round, echo, pacing, and what was measured.
Questions, the container, the server Provider questions and pages that fill in, playback in full, repackaged streams, the server's paths, hosts and media.
Errors, routes and restoring Status codes, the routes for sharing and backups, and how a restore is applied.
The gate The Android app's door in full.
Plugins in detail Refresh and the Home layout, the sandbox's network rules.
Frontend notes The router, the player's quiet and stuck watch, the watch party's speed.

The web build

Code: cmd/mantle/main_js.go, internal/core/browser_js.go, frontend/src/web/ (core.js, worker.js, files.js, seal.js), frontend/src/bridge/persist.js, frontend/vite.config.js, frontend/measure/closing/closing.mjs, BenchmarkSave and BenchmarkMarshalState in internal/core/forward_test.go.

The web build runs the core in the browser. cmd/mantle/main_js.go is the same program as the others, built with GOOS=js GOARCH=wasm, and core.InBrowser (internal/core/browser_js.go) is true in it.

  1. main.js sees the Vite mode page and awaits startCore (src/web/core.js) before anything asks the core.
  2. startCore registers the service worker as sw.js?page and waits until it controls the page. A page that was loaded before the worker had it (the first visit, a hard reload) then posts keep with the addresses it loaded (performance.getEntriesByType('resource')), and the worker keeps the page and its own files among them that it does not have yet, so the next start needs no connection.
  3. It takes the Web Lock mantle-core. Held by another tab, it asks "Mantle is open in another tab." with Use it here, which posts yield on the mantle-core BroadcastChannel; the holder saves, stops its worker, lets the lock go and offers it back. The holder listens from the moment it has the lock (a BroadcastChannel does not queue messages), and answers a yield that came while it was still starting once the core is up. A core that fails to start lets the lock go too.
  4. It starts a module worker (src/web/worker.js). The worker sets globalThis.fs to src/web/files.js, a file system in memory that answers as Node's fs does, which is what Go's syscall/fs_js.go asks. It reads the files back from IndexedDB, loads wasm_exec.js and runs the core. Each change is saved to IndexedDB a moment after it is made. The core's secrets.dat is kept sealed there (src/web/seal.js): AES-GCM under a non-extractable CryptoKey that the page generates and keeps in a database of its own (<files>-keys), sealed as files.js saves it and opened as it loads it, so the core sees it as it is (internal/secrets does not seal in the browser: protect_js.go). What is sealed is secrets.dat and every file named for it (covers: the secrets.dat.tmp the core writes beside it before it renames it, and what it sets aside), so none is stored in the clear. A file kept before is read as it is and sealed by the next save; one that cannot be opened with the key (another's, or none: lost, or the storage copied without it) is left as it is kept, for the core to set aside, and is not sealed again; a key that cannot be looked for (the keys' database blocked) is asked for again a few times and then the load fails ("Mantle could not start"), never taken for a missing key; a seal that fails, or a browser with no WebCrypto, fails the save, and nothing is stored unsealed. A save reads every file it stores before it seals any (nothing is awaited between), so what one transaction holds is of one moment: the core goes on writing while a seal is made, and the state would otherwise be older than the sync state beside it. This protects against a dump or export of what IndexedDB holds, not against script in the page, nor against the browser's profile folder (the key is kept in the same place on the disk). A transaction that fails (no room left) makes flush reject, and what it was to write stays dirty: a state change is still answered as done (it is applied), and the worker tells the page the notice "Your change isn't saved yet: . Mantle keeps trying." beside the answer (core.js toasts it, as the timer's notice, so one notice shows); requests that only look or poll (often in worker.js) are not saved before they are answered, and a burst of changes is one save, 50 ms after the last, and a retry is set after a wait that doubles at each retry that fails (up to a minute), until one works; changes made meanwhile are saved with it. KeepLog keeps no log file in the browser, which has nobody to read it: it would be rewritten whole at each save as it grew.
  5. cmd/mantle/main_js.go starts the core at /mantle-data, sets mantleServe, mantleCancel and mantleSave on the worker's global object and calls mantleReady.
  6. The service worker answers a request for /api/ or /art/ by posting it to the page that made it, with a port. The page passes it to the worker, which answers it with mantleServe: the API's handler, run on a recorded request. An aborted request ends its context (mantleCancel). A redirect is answered with Response.redirect.
  7. The page asks the worker to save (mantleSave, then the files) when it is hidden or goes, each ask with a number that the worker's reply (saved) repeats, so each wait ends with its own reply (and with the worker, when it is stopped). A request that changes state is saved the same way before it is answered, but the ones asked again and again (often in worker.js: prefetching, the log, the display, a playback's place) are not: the core saves itself each second it has changed (Persist). A save that fails is said to the page, which shows a toast.
  8. cmd/mantle/main_js.go answers 500, and logs the panic, for a route that panics, as net/http does for a request: nothing in the worker would otherwise stop the whole core from ending.
  9. The page asks the browser to keep its storage (keep in bridge/persist.js, which calls navigator.storage.persist(), as the core starts; the browser may refuse). Settings → Data and storage → Keeping your data says the answer (keeping, navigator.storage.persisted()) with a button that asks again, in the web build only (Safekeeping in app/pages/settings.js). The Android app's page says there, in the same group, that Android does not back its data up (inAndroidApp). The save as the page goes (point 7) does not finish when the tab closes, and nothing sound can make it. Measured in desktop headless Chromium alone (frontend/measure/closing/closing.mjs: the web build's page, worker and files, files.js itself, with the core stood in for by a save that marshals for 36 ms, what a save (BenchmarkSave: marshalled and indented) takes in WebAssembly in Node on the development machine for a 516 KB state, 34 to 42 ms, and 8 to 10 ms natively, then writes through files.fs), 15 runs each: a save asked as the page goes (pagehide, as it closes or navigates away) reached the database 0 times in 15, the handler having run every time and the worker having got the message: the browser ends the worker before the save gets past its first steps. A save asked in a page that then lived on took about 43 ms from the message to the database (47 at most): it finished 15 times in 15 when the page lived 50 ms more or longer, and 0 in 15 at 25 ms or less. So the save when the tab is switched away from finishes where the page goes on running, as a hidden tab does in a desktop browser; a phone's browser may freeze or discard a hidden page, which was not measured. Nothing makes the close certain: the state is the worker's memory, which the page cannot read, and a service worker, which outlives the page, has none of it either (posting it the files already written, as the page goes, would cover only the files' batch, with a second writer of the database). What is at risk is what changed since the last save: the core's own saves come a second after a change (Persist: it marshals the state each second to see if it changed, 5.5 to 5.7 ms natively and 32 to 36 ms in WebAssembly for a half-megabyte one (BenchmarkMarshalState), which is why it is not made more often), the files' batch (wait in files.js, 50 ms: it was 300, which left a change in memory for 300 ms and more: of a core save in the files, a page closed 350 ms after kept it 15 times in 15 and 200 ms after 0 in 15, where with 50 ms it is 15 in 15 at 100 ms) after that, and the page's before it answers a request that changes something. Another browser would change the milliseconds, not this: by the HTML standard a page's dedicated workers end with it, so in none of them can the page or a service worker make the close certain. A file a newer version wrote (something kept) adds the graft to a save (above: under 2 ms in WebAssembly, wherever the field is, to the 34 to 38 ms a save takes), so a save that is asked takes up to about 49 ms in the worst run measured: a page that is switched away from and goes on for 50 ms or more still has the time, with 1 ms to spare, so the margin is thin, and a phone's browser may freeze the page sooner. The page build's Content-Security-Policy is a tag that vite.config.js puts in index.html in mode page only (a static host cannot send headers). It lets the site's own scripts run, with wasm-unsafe-eval for the core, and the page fetch from any http: or https: address, load images and media from anywhere, and frame YouTube's trailer player (app/youtube.js). With upgrade-insecure-requests it asks for an https: address in place of an http: one (but this computer's), so nothing on the page comes over plain http and no browser calls it insecure; mk web -serve, whose page is on http://127.0.0.1, leaves that out of the page it serves. The page has no inline style: a style attribute in markup is refused (Preact sets styles through the CSSOM, which is not), so a drawing in ui/faces.js uses classes.

What a page cannot do, the core does not try:

  • Network (browser_js.go) is the browser's fetch, through Go's net/http, without the core's User-Agent: a page's own makes the site ask first, which most refuse.
  • A plugin whose manifest says native (its service refuses web pages) does not run (plugin.Plugin.runs), is seen as off, and cannot be turned on (ErrNeedsApp). What it was set to is kept.
  • Download answers ErrNeedsApp, and Downloads is not a section.
  • Artwork is answered with a redirect to where it is (ArtAddress), since a page may show any picture but read only some.
  • The kept answers may take 16 MB, not 64 MB, because the files are in memory too.

The frontend asks inPage() and downloads() (core/host.js) to hide what does not apply, and runsHere(p) for a plugin.

What start does

Code: cmd/mantle/app.go, internal/portable/, internal/secrets/, internal/core/persist.go, folders.go.

Before start, cmd/mantle/main.go (the window; once Wails's own single-instance check has handed a second run's link to the first) and cmd/mantle/web.go (before it binds its port) take the data folder with portable.Lock: an exclusive operating-system lock on <data>/mantle.lock (LockFileEx on Windows, flock elsewhere, nothing in the browser), held for the life of the process. The system lets it go when the process ends, a crash included, so a stale lock never blocks a start and a thumb drive moved between computers never starts locked (a pid file would). Lock tries for lockWait (5 s, as the first run may still be saving on its way out) on a held lock (EWOULDBLOCK or ERROR_LOCK_VIOLATION, held), then gets ErrInUse, which the run logs to logs/mantle.log (refused) and ends (the window tells it in a dialog, shown once the app's loop runs, as it has no console). Any other lock or open error (read-only media, no lock support) runs unlocked, with a warning that Lock returns for the caller to log once the log file is open (start opens it after the lock), so a thumb drive that is only readable still works. A second run ends: two cores on one folder would each save over the other, or one apply a restore under the other. An update's replacement program waits for the old one to end first (awaitReplaced), so it finds the lock free.

start in cmd/mantle/app.go builds the core in this order:

  1. Creates it with core.Empty (a new install: one profile without a name). With MANTLE_SAMPLE=1 and a build that has the sample, it uses core.New.
  2. Keeps the log in <data>/logs, and sends Go's own log there too.
  3. Adds the plugins that come with the app. They are embedded from plugins/*/*.lua. With MANTLE_PLUGINS set (in a build without the production tag only: build.DevEnv), they are read from that folder and reloaded when their files change.
  4. Unless the sample is in use, calls UseRealProviders: the sample's stand-ins go off, and the real plugins that need no setup go on.
  5. Opens <data>/secrets.dat as the store for plugins' secrets, applies a restore waiting in the data folder (ApplyRestore: see Restoring), and gives the core the network. The file is sealed under the data folder's own key, secrets.key, made the first time and never changed: the folder is the profile, so a copy of it opens as the original does (internal/secrets/seal.go, bound). The Android app's window gives its Keystore's key instead, over the program's standard input (-secrets-key-stdin, secrets.UseKeyFrom, Secrets.java), and no key file is made: with the flag set, a key that does not come or is empty stops the program, and the Android build does not start without the flag, so it never falls back to a key in the folder. At each start the files set aside (.unreadable-*, listed by name, not as a pattern) that can be read are added to what the file has, where it has none (takeIn), and renamed .adopted-*; a file kept before (plain, or a Windows user's DPAPI) is read and replaced by a sealed one, staged and read back first, the old file moved aside and overwritten and removed (replace, wipe; flash storage may keep the old bytes), and a plain file taken in is sealed under its new name. A sealing that fails is logged and tried again at the next start; for a file that could not be read, the store is that run's alone and the file is left where it is. A file that could not be read gives ErrUnreadable, and with secrets taken in ErrRecovered (the page says so: "recovered"). A secrets.key that is not one is set aside; one that cannot be read is tried again, then the page says why (KeysUnread: "aside", "unsaved", "key").
  6. Keeps providers' answers in <data>/cache/answers.
  7. Gives the core its home (the folder that holds the data folder: the program's), for relative download folders. A path is kept as storedPath (folders.go) says, the one rule for every path: as typed if typed relative; relative if given in full but in the home or under it; else in full, as given (a path from a drive's root, \Films, is on the program's drive); text that cannot be a folder is refused. A folder kept in full must itself be there (freeFor, online, thereAt): one that is not shows as not connected, and none is made on a drive that is there, not by a download that was already in it (it waits: pumpDownloads, placeFor), a first download before the folders were looked at (roomIn), a backup copy, the details beside a file (describeOnDisk), or Open folder (FolderPath; makeBelow). Adding a folder or changing its path makes it (make), after everything is checked.
  8. Adds the plugins installed from links, from <data>/plugins, and watches them for changes.
  9. Gives the core the media address (each shell supplies its own).
  10. Reads <data>/mantle.json. If the file cannot be read, it is set aside as mantle.json.unreadable-<time>, and the copy that loads is opened instead: a newer version's copy of what this one saved last, then mantle.json.bak. Only when neither loads does Mantle start afresh. Then it saves the state whenever it changes.
  11. Refreshes the plugins, starts describing the profile's titles in the background, and resumes downloads.

Saving, loading and going back

Code: internal/core/persist.go, forward.go and its tests, folders.go, profiles.go.

A save that fails (Persist's every second, and Save as the app ends, which tries saveTries times) is kept as saveErr (shown: the cause, with no folder path), carried as Downloads.SaveError on the downloads long-poll (c.dl.change() when it changes, so the banner in App.js follows at once and no timer asks), and logged as it starts to fail and as it works again, not at every try. The saver tries again at each tick, as what it last saved whole (last) is only set by a save that worked; the file beside is never half-written (write renames), and stop does not exit with an error when the last save fails: it logs it and leaves the last good file. Downloads keep keepFree on the drive of the folder they write to, which is the room Mantle's own saves need when that drive holds mantle-data: roomIn counts what the downloads running on the folder's drive (driveOf: its volume on Windows, its device elsewhere, kept in diskState when the folder is looked at) have still to write (pending, an unknown size counting as the reserve) when a new one picks its folder, and fetch.Job.Room is asked before a job's first byte and at each sync of a running one (every 32 MB), ending it with ErrFull ("The drive … is on is full") below the reserve, whatever the size its source gave.

How syncing is going is the core's state, said in words. rounds (sync.go, behind a lock of its own, so Syncing never waits for a.mu, which Household holds across its requests) keeps the last whole round's trouble (in plain words: together.Words, the one place the service's codes and the network's failures are given Mantle's words, which the server's replyOr uses for every Together answer too), detail (what Details shows: the status and code the service answered, or that it could not be reached, never Go's text with the service's address, which only the log keeps), synced and failing, when rounds began to fail in a row; a round that goes through clears them. Only whole rounds say (Core.failed): a push alone, as a page put out of sight or a quit makes, would clear a failure the next whole round finds again, and a round that something else ended (the caller's deadline or leaving, the account changing) is no failure and changes nothing, while one that runs out of its own roundEnds (errRoundLate, its context's cause) is: a service too slow to finish a round is failing. A device taken out of its account (ErrRemoved: signed out in the round) leaves nothing failing, and signing out forgets synced too, so a new sign-in says "Not synced yet" until a round goes through. Household carries them (and unread, apart: why its own read of the account's profiles and devices failed, which is not syncing failing), and GET /api/household/syncing (Core.Syncing) alone, from the core with no request to the service, for the top bar; the page's Sync group, its badges and the bar all say it from failing. Core.failed bumps the news when failing starts or ends, and Syncing says stalled once it is two minutes old by the core's clock: the app asks again on news, and every 30 seconds (a local request) while a failure has begun but is not yet stalled.

The state and the sync state are of one moment. A device keeps two files, mantle.json (what it has) and together.json (what it knows it synced: syncState), and a record that the second lists as synced and the first lacks is read as deleted here and pushed, for every device. So the sync state is only ever written after the state is saved, in the one place it is written (account.save): it calls the core's saveForSync first, which saves the state (saveTo, with the saving lock, one try: the caller holds a.mu, which a wait would hold up, and the state's own save tries again at its next tick), and writes the sync state only when that worked. When that save fails the sync state is not written (it stays as it was, which only has more pushed again), and a round pushes all the same: the account has it, and the older sync state does not claim it. A round is not the only writer (a profile made local, a household change, a rekey, a restore); account.save is, and nothing else writes the file. The state's own save, once a second, shares Core.last (what was last saved) with saveForSync, so the state is not written twice, and account.save writes only when the sync state changed, so a round that changes nothing saves nothing. Lock order: saveForSync takes the saving lock and the core's lock inside a.mu, so a caller of account.save does not hold the core's lock (LeaveProfile lets it go first). A caller that must not go on without the save, such as a rekey that marks what is to be finished (rekeyOnce), stops on its error.

Both files carry the same pair (Core.pair and syncState.Pair, omitted when 0): the id of the moment the records last moved here. It is made again (newPair) where the account's side moves, and nowhere else: as a pull that has records begins to apply them (applyPulled, under the core's lock, before the first is applied) and as a push is sent (push: a push of one batch once it went; one of several batches, a first sync of a big library or a restore, as it begins, saved at once with everything still queued (account.save), and not again per batch, so the state's own save writes the same pair while the batches go and a device stopped between two starts with the files in step and the rest queued; a save a batch would write both whole files each time, quadratic in the push: 20,000 titles, 81 batches, 7.3 s against 0.5 s). It is 64 random bits, not a count: a count starts again from a state put back, and meets a number that the other file, of another moment, already carries. It is set in memory (the state and the sync state's, both), so a state saved by anything after that, the state's own save at any tick in the middle of a round included, is of an id the sync state has not been written with, until account.save writes it. A save that moves nothing keeps the id (a change made here, a household change): a stop after the state's save and before the sync state's write leaves the files of one moment, and what the state has that the sync state does not know of is this device's own changes, which the next round pushes as it ever did.

The id is checked once both files are read (checkFiles, at the end of Persist (after its copies of the file as found) and of useRemote, whichever comes second): when they differ, or the state's write is the lower, the sync state is not of the state's moment, whichever is older (a stop between their writes in a round that moved records, or in the middle of one; a copy of one put back, a disk image, a sync tool; mantle.json.bak over the state; the state missing), and every synced profile gives way to the account (yieldAll marks each book, profileState.Yield, and giveWayAll has each of them give way as checkFiles ends, once both files are read: account.giveWay knows every record here as set at no time and forgets what the state lacks, so what differs is taken from the account and nothing is pushed as deleted, and a record the account deleted is not stamped newer by this device's restart), the way it does when the state was opened from its copy or restored; the household's book gives way there too (giveWayAll makes its records with c.shared().records(): a build adds the plugins and their keys before the account, app.go, so they are all there, and AddPlugins and UsePluginDir refuse (errPluginsLate) once the give-way has been made (account.gaveWay); only a state of the service this build reaches is given way to, since useRemote leaves another service's be); the sync state takes the state's id first, so that what it writes next agrees with it, and a write of it that fails leaves the files differing, to give way again. What giving way costs the device is what it changed since its last round and before it started: the account's version of a record replaces this device's (yield), so an edit made here then, and a removal of something the account has, are not kept; what the user changes after the start is theirs, since it is known as set at no time and the look stamps it as it does any change (a device started with no network gives way at its start, not at its first round, which may be hours on: #340; the household's book too); what the account lacks is pushed, and is kept. Files saved before the id (or the number) was kept have none, and are alike; a build before it keeps what it has no place for (forward.go), so it writes the id and the number back as it read them and the files still agree when this build reads them again, and this build reads theirs as it always did. Such a build that stops between a round and the sync state's write is not seen here (it never moved the id), so the first start after a downgrade and an upgrade has no more protection than it had.

What the id cannot tell is a state put back from before a save that moved nothing: a look that stamped a change writes the sync state after the state with the id as it was, and a copy of the state from before (the one kept beside it, say) has that id too, and would take what it lacks of the stamped change for deleted now. So each write of the sync state is numbered as well (writes, in both files: saveForSync counts it as it saves the state for it), and a state with a lower number than its sync state is older than it, and is not of its moment either (#352). Such a state with the same id gives way only where it differs from its sync state (giveWayAll, by looks): no record moved between the two, so a record whose value in the state is the one the sync state knows is of one moment in both, and keeps its stamp and state (a change made here and not sent still goes up at its stamp, and wins over an older one made elsewhere); one that differs, or that the sync state does not know, gives way, and what the state lacks is forgotten, not taken for deleted. That is kept with the mark to give way (YieldLooks, beside Yield), so a start stopped before the give-way makes the same one at the next; and until a sync state is written with the mark, the state is saved at its own number and the sync state at its own (asIs), so a stop or a write that fails within those saves leaves a state that is older still (one saved a number past, and with the sync state's write failing, was taken for of its moment). A write is numbered past both files' numbers, so no number is used twice: a copy of the state from any time before has a lower one than the sync state (bringing the sync state's down to the state's, a first answer, began the count again, and a newer copy put back after read as of its moment). A count does here what it could not do for the id: what it tells is which of the two is older, and a state put back only ever has a lower one; once both are read the number goes on from the higher (checkFiles). A sync state with the lower number is the other way round, a stop between the two saves (or that file put back), which the id tells when records moved; when none did, what the looks between stamped is stamped again at the next look, later than it was made, so a change made elsewhere in between can lose to it (#319's kind; not closed here).

A second line, for what the pair cannot see (both files put back together from before the number, say): when the state is read, the records a profile's sync state has as synced and the state lacks (holdLost) are braked if they are brakeMin (100) or more and more than the profile has left, or brakeMax (250) or more whatever is left (only what changedHere would delete is counted: kinds that are never taken for deleted, recordKind.stays, are not, nor are the own records of a list that is here and shared, ownShared; the lists of a build before the one-record-per-list format, list/<id>, are, and upgradeLists leaves a braked one as it is until the user decides). A braked record stays known as synced and its name is kept in profileState.Braked (in together.json, so a restart holds it too, until the user decides, even if more is added meanwhile); changedHere skips it (so the look, stampHere, which stamps with it and keeps what it stamped, does not stamp a held deletion either), so no deletion is made; and one changed here and not pushed before the files lost it (queued still) is not pushed while they lack it (pushRound), for it would go up as nothing, a deletion said to be meant, which neither brake would stop, nor counted as waiting to go up (queued), so the loop is idle while the brake holds. A record a pull applies, one that is here at a start (holdLost), or one a look finds here again (changedHere: the person added it while Mantle ran, so taking it out again is theirs, and goes up) is no longer braked, and a mismatch's yield clears them all (checkFiles has the give-way made first and holdLost after, so a book that gave way has nothing lost to brake). It is logged, and counted in Household.Held. Settings → Account shows it as "Deletions held back" (account/held.js, and a line beside Sync now), with two choices, routed like any Household action (POST /api/household/held/confirm and /held/putback, in the one route table every shell serves; the web worker keeps the files after them as after any change). ConfirmHeld (Delete them everywhere, asked first) clears the braking and has changedHere make the deletions now, saved at once, so a restart before the round does not brake them again; PutBackHeld (Bring them back) forgets the records and reads the account from the start, so what it has comes back as the pull reads it. Only what is gone as the state is read can be braked: a deletion made while Mantle runs (a list of 600 titles, the whole history, a profile) was not found by it, so it goes up at once, whatever its size; one made before the app stopped, that is most of the profile, or 250 records or more, is braked, and asks. Shared lists and the household's records are not looked at (only the pair protects them). The shell asks the core how many are held (GET /api/household/held, local to the core, no request to the service) when its pages show (it starts, a PIN is given, a profile is opened) and when the core has news, and says so once in a run, in a toast with Review (app/held.js).

A data folder has one sync state and one session, whatever the channel (together.json, and @together in the sealed secrets: Core.keys). A release and a beta reach one service (together.Backends), so a folder they open in turn is one account's, and each build carries on from what the other left: nothing is carried between them, and nothing is kept apart. A folder signed in to another service (dev's, a household's own) is left be (useRemote, Household.Away).

A folder whose beta once crossed to staging needs no code of its own. The builds between 0.3.1 and 0.3.2 gave betas a second account there (#349), which #402 removed: of what they left, the second sync state (together-staging.json), the session under @together staging, the beta's own friends (Profile.Beta, kept in the file as any field this build has no place for) are not read, shown or sent. Two fields of the state are read as those builds wrote them: writes, a number for each sync state by file, of which together.json's is taken (writesOf), and pairs, read only so that it is not kept and written back stale (such a build, opening the folder after, reads pair then; it cannot read writes as a number, so it takes the state for older by looks, once). If a release saved the state last, its pair and its write's number are together.json's, so it carries on with nothing given way: what it had saved and not looked, and what it had queued and not sent, go up (TestFolderOfABuildWithWritesByFileKeepsWhatItHadNotLooked, …SendsWhatItHadQueued). If the beta saved it last, its pair is the staging file's: the files are not of one moment, and every book gives way to the account (checkFiles), so production's value wins, nothing production has is deleted, and what only the folder has (a title added on the beta) is pushed. A change or a deletion made on the beta that no release carried is as production has it, and so is one the release itself had queued in together.json and not sent before the beta ran: the give-way is whole. A block made on the beta goes up with the profile's records, and the round has production carry it out (tellBlocks) (TestFolderABetaCrossedOnOpensWithTheReleasesAccount, TestFolderAReleaseRanLastOnAfterItsBetaCrossedCarriesOn, folder_test.go).

Removing a profile (RemoveProfile) takes its private download folders with the downloads in them (what is finished stays on the disk, as for RemoveFolder; what is not has its partial file removed), passes the finished downloads it made in shared folders to the profile that is then current, and cancels the rest: no download keeps an owner that is gone. The sources kept for its titles (under ~<hash of its id>/, kept between runs with the providers' answers) go too (forgetSources). The state before it is kept as mantle.json.before-removing-a-profile-<id>, one for each profile, the last keptRemovals (keepLast).

  • Profiles. Each has its library, lists, favourites, progress, watched marks, history, search history, settings and the source it chose for each title. One profile is current. This is a single value for the whole core: every client of one server shares it.
  • The registry of titles that providers have described. It is saved with the profiles that refer to them; the titles nothing refers to (a profile, a download, a watch party, what is playing) are kept at most maxTitles of, those kept longest ago going at each save.
  • The plugin host and the plugins' loaded scripts.
  • The downloads and download folders, shared by the device's profiles.
  • What is playing, if anything.
  • The log (the journal), shown in the Inspector and kept on disk.

mantle.json holds the profiles, who was watching, the plugins as installed and set up (with what each was granted), the catalogs, the download folders and the downloads. Providers' answers are not in it.

A version that goes back reads what a newer one wrote without losing most of it (forward.go). decodeKeeping reads a file into its Go type and keeps, in a tree shaped like the file, what the type has no place for; graftBytes puts that back into the type's own JSON (compact, as json.Marshal writes it) by insertions: an unknown field goes in at the start of its object, so the scan goes only as far as what is kept goes and the rest of the bytes are copied as they are. The file is indented after, as it was. Exactly what is kept, and what is not:

  • In an object (the file itself, a struct, a map's value): the fields this version has no place for.
  • In a list of objects: the same, for objects the list's type tells apart by the fields tagged keep:"id" (a profile's, list's, plugin's, repository's and folder's ID, a download's Key, a title's ID, an episode's Key, a plugin setting's ID, a title folder's file's File, and a play in a history by its title, episode and time together; an id is text, a field of another kind is none), and only where that value is in the list once as it was read and once as it is written (two plays alike keep nothing, and one of two removed here does not hand its field to the other). A list of another type (a list's rules and members, a feed's releases) keeps nothing, and that is the design: its objects are told apart by nothing (two rules can be the same, and members and releases are the account's and the feed's, made again), so a field kept for one would go to another's. TestEveryListSaysWhatTellsItsObjectsApart makes a new list say which it is.
  • A field this version has a place for but cannot read (another type) is not kept. It costs that field, not the file: it is left out of what is loaded, this version's own value for it (its default) stands and is what is written, the paths are in the log, and the file as it was is kept beside it as <file>.before-unreadable (keepUnreadable: the last one, for mantle.json whichever file it was loaded from, the copy that opened included, for together.json as it loads, and for a title folder's mantle.json as it is written over), so what the newer version had there can be recovered by hand. What one such field costs is its value: a setting back at its default, one list's titles, one profile's favourites, an entry of the progress. A counter that could not be read is 0, and a new id skips any in use (nextID: profiles, folders, downloads, repositories).
  • What the user made is not left out for a default. In mantle.json, a top-level field that holds it (the profiles, plugins, repositories, folders, downloads and titles: keep:"strict") of another type, an object of one, or one of those objects' own fields (a download's key, a folder's path or id, a profile's ID, its settings or favourites as a whole) is not read, for folders and repositories would silently become the defaults and a download without its key is dropped; what is deeper (a setting inside the settings, a list's titles, a progress entry) is left out as above. So is a file of a higher version. Such a file is what a newer version wrote (as far as a file that is JSON all through can say), and the saved state is then treated as one that is damaged, with its own words (errNewer): the file is set aside as mantle.json.unreadable-<time>, never written over, and what opens is the copy that newer version kept of the state it found (mantle.json.version-<its build>, named by the build the refused file says: an older Mantle's last file, in the older one's shape), else mantle.json.bak if it reads (the newer version also makes it, and may have written it in its shape), else Mantle starts afresh, and says in the log and to the person that it is a newer Mantle's (Recovery.Newer). So a newer version that changes what an existing field holds, rather than adding one, costs an older one its start from where it left off (the newer file is kept, and goes back in the place of mantle.json when the newer version is used again); a field added is kept.
  • What is kept goes with its object. It is not changed. An object this version updates (a list it adds a title to) keeps what the newer version attached to it: those describe the object as the newer version wrote it, a newer version that reads it back sees its own fields, and dropping them at each update would lose more. What is kept of an object that is not there is dropped, so an Undo of its removal puts the object back without the fields a newer version added to it, and an object made again (a plugin installed again, a title's progress played again) has none of the old one's. A parent this version omits when it is its zero (the updates: omitzero) is written with just what was kept of it. What a device knows of an account's records (profileState.Known, and the list states, which read themselves) keeps nothing, since what a newer version adds there is about a stamp that has since changed, and what was kept of together.json goes with the account on every way it is left (account.forget, joining another, deleting it, leaving another's).
  • Synced records are not kept inside, by design. A record this version makes is read into this version's type, so an edit of it here pushes it without the fields only the newer version had. A record is the value of its kind, replaced as a whole by whoever writes it last, so what a device does not know it cannot say; keeping them would be a second copy of every record in the state, and a device that wrote them writes them again with its next edit of the record (TestARecordOfAKnownKindIsReadIntoThisVersionsType).
  • A newer version's records are left alone. A record this version would never make by that name is a newer version's (subject.foreign, sync.go). It may be of a kind this version lacks, or of a kind it has by a name it does not write: a setting it has no field for (setting/<field>), a part of a list it does not keep (own/<list>/<part>). That is what a newer version adds most often. A beta and a release open the same folder and the same account, so a version meets such records at the account and in its own sync state all the time. It has no value of one, so it cannot tell a deletion of it from its not being here. What it makes is said by name: knowsRecord (a profile's), knowsShared (the household's), listKind (a shared list's).
    • A pull neither applies one nor notes it as known (applyPulled).
    • A look and a round do not take one the sync state knows for deleted (missing), send it (pushRound), stamp it again (restamp), hold it by the brake (holdLost) or count it as waiting to go up (queued).
    • A service restored from a backup is sent nothing of one (pushBack); the newer version's device sends it again when it finds the service behind.
    • A restore does not put one back (changing).
    • A rekey of a shared list reads the list whole, once, and sends its foreign records on under the new key with their value, stamp and writer (rekeyOnce). They go first, in the page that replaces the key. Its limits: a rekey cut short keeps only those that fit that first page (500 records, 8 MiB); a record that does not open with the list's key cannot be sealed again and goes with the old key's (the log says how many); and a record that was its writer's alone is no longer so, as a pull does not say it was.
    • The cursor moves past one, so where it came is kept (profileState.Passed, the push before the first passed over, and PassedBy, what the version that passed it over makes: madeHere). A version that makes other records pulls the book again from there, once, as its next round begins, and applies what it makes by the stamps. So a beta that takes turns with a release on one folder has what other devices changed or deleted of its own records while the release ran. A version that passed them over itself does not pull again for them. A rekey marks the list from its first push, as the foreign records went up anew.
    • What a version before this one noted of such a record as it pulled it (applied as nothing, so known as gone at the record's own stamp) is forgotten at the book's next round (forgetForeign), so a later version that makes the record does not take the note for a deletion synced. Only notes with another device's stamp are: a deletion a newer version made on this device has this device's stamp and is that version's to keep.
  • A record of a kind that stays that is no record here (a plugin of the household that waits to be installed, a profile's choice of one) is known as it came, not as gone (takePulled), and is never sent while it is not here (pushRound): there is no value to send, and a service restored from a backup would be sent its deletion.
  • Export files are read whole by Import and never written back, so nothing is kept: one with a field of another type is refused as not an export (as it always was), one a newer version made (mantle: 2) is refused, and a field a newer version added that this has no place for is left out.

It covers mantle.json at every level its types have (so the plugins as installed and set up, whose values are read as they are, and the plugins' kept data, which is raw), together.json and a title folder's mantle.json. The copies beside the state are what they were saved as: .before-* as it would be saved (with what was kept put back), .bak, .version-* and .before-unreadable the file as it was.

What it costs. A file with nothing to keep is read by the typed decode alone (DisallowUnknownFields: no type that reads itself hides a field from it but time.Time, json.RawMessage and the lists' states, which keep nothing, which TestEveryListSaysWhatTellsItsObjectsApart checks); a file with an unknown field, or one of another type, fails that and is read again as a tree, once, as it loads. A save marshals as before, under the lock (compact: the file is indented after, so a tick that finds the state unchanged does less than before), and compares it with the last; the file is made, outside the lock, only when the state changed, and what was kept is put back then. Measured on the development machine (a busy one: ranges over three runs; WebAssembly in Node, two), a state of 516 KB as the file has it (3,000 titles with their progress: BenchmarkDecode, BenchmarkGraft, BenchmarkSave, BenchmarkMarshalState):

native WebAssembly
load, nothing to keep 7.5 to 8.6 ms, 3.8 MB, 17,214 allocations 30 to 54 ms
load, a plain Unmarshal (what it was) 7.0 to 7.3 ms, 1.7 MB, 17,197 allocations 36 to 46 ms
load, a file with an unknown field 35 to 39 ms, 12.5 MB, 150,000 allocations (once, as it loads) 170 to 240 ms
save, marshalled and indented (what it was) 8.5 to 9.6 ms 34 to 38 ms
save, nothing kept 7.9 to 8.6 ms 32 to 39 ms
the graft alone, a field at the top of the file 0.07 to 0.3 ms, 21 allocations 0.5 ms
the graft alone, in a profile 0.35 to 0.47 ms, 44 allocations 1.1 to 1.3 ms
the graft alone, in a title's progress (the first, or the last of 3,000) 0.37 to 0.49 ms, 64 allocations 1.4 to 1.9 ms

So a load with nothing to keep costs a little more than a plain Unmarshal did: natively about a tenth more time and twice the bytes allocated (the decoder's own copy of the input), with as many allocations; in WebAssembly the difference is inside the noise of the measurement. A load of a file with something to keep costs several times that, once. A save with something kept costs a save and the graft on top, which is under half a millisecond natively and under two in WebAssembly wherever the field is (the graft reads no more than the way to it), under a twentieth of the save. The same state saved again (the close; the web build's saves before a request is answered) is grafted again and costs the same.

Other parts of the core

Code: the files named in each item; the package comment of internal/core/core.go maps them all.

  • Importers (importers.go): an importer plugin gives a profile's library, watched marks, places to resume and lists. The core merges them by fixed rules (nothing removed, importing again changes nothing).
  • Settings (settings.go): a profile's preferences. The page carries defaults (what each setting starts as, the same for every profile) and defaultShelves (Home's shelves in the order they start in), so the page can tell what has changed and put it back with an ordinary PATCH /api/settings. A shortcut key goes back by an empty key.
  • Setup (setup.go): a plugin's settings, its account sign-in (a code the user enters on the provider's site, which the core follows) and what it still needs. requires is what a plugin must have set up to work: the settings it marks required and gives no default, in their order, then signing in if its account is required. needs is what of that is not done yet. The Plugins page carries both (requires for every plugin, needs for those that are on), and a plugin's setup carries its needs. The Plugins page, the first run and the setup form read them and do not work them out again.
  • PINs (pin.go): a profile locked against the others of a household. See below.
  • Share (setupshare.go): the setup as a code or file, and bringing one in. See below.
  • Sorts (library.go: SetSort): how the profile sorts each list (Profile.Sorts, by list id; all for all titles; a sort, with - after it when reversed) is the profile's own choice, a shared list's too, and one sync record each (sort/<list>; an older build ignores them and keeps them as the file had them). A Library query without a sort is sorted by it, and the answer says which (sort, sr). A list's deletion drops its sort, and a shared list's new id (renameList) takes it along. The Library's search and filters are not the core's: they are the profile's on this device (useProfileKept), one for every list.
  • Files (export.go): core.File is a file the core makes (an export, a shared setup) with its name and bytes. File.SaveIn(dir) writes it into a folder, refusing a name that would land outside it, and answers its path.
  • Onboarding (onboarding.go): the first run's step, saved.
  • Downloads (downloads.go the queue, downloadrun.go fetching one, downloadrow.go the page rows, folders.go, with internal/fetch). A title's files, details (mantle.json) and pictures are in a folder of its own, named by onDisk (ondisk.go). Two titles can carry one IMDb id, so a title that finds another's files in the folder it would take (in the list, in any download folder, since a backup folder copies each file to the same place; or by the details beside them) takes its own id into the folder's name as well, and keeps that folder for its later files (apart). Each finished file in the details says its title (sideFile.ID). findDownloads uses it for a title the app knows, else the folder's title (the details hold that one alone), so a folder two titles shared (made before) is found again as the title that last wrote its details, but for files finished since. A download's file is only served (MediaFile, DownloadPath, the pictures beside it), copied to a backup or deleted when it is inside its folder (place.inside, folders.go): its name is local (no .., root or drive), whatever the record says. A link the user made inside the folder (a junction or symlink of a title folder) is followed, as it was placed: the threat is a record's name, not the user's own folder. A copy of a finished download counts unless it is cut short: smaller than the size the download was recorded at, told by a stat, never a read (place.there). Damage (a full drive, a drive pulled out, a write cut off) only cuts a file short, and nothing Mantle writes ends bigger than its record; a bigger file is whole (a record of a build before 0.3.1 kept the link's estimate when the host said no size, which was never more than the file). copyFile does not copy a file smaller than the record, copies one as big or bigger at its own size, and keeps no copy that comes out at another (the file changed as it was copied). LookAtDownloads, finding a download's own file bigger than its record, records it at its size and writes the details beside its copies again, so its backups are held to that size; not if a backup is there at the recorded size, a whole copy of what was downloaded (its own file was changed by hand, or is being written yet by a copy that sets the size first): the record and that backup stay, and its own file plays. A backup cut short is dropped from Backups and copied again, over it, when the download has a whole copy to copy from (with none, it stays among its backups, where its copies are); each download's copies are judged by its own record, so two downloads copied to one place in a shared backup folder settle on the bigger file after one copy. A download with no whole copy is missing (dl.missing says why: gone, or cut when its own file is there, cut short, which its row says as "Only part of this file is in the folder …"); retried, one cut short downloads again in its place, its .part renamed over the file once whole, and its backups' copies made again over theirs. Until it is whole, Download.Replacing names the folders whose old copies of its name it replaces (its own, its backups'), so one removed before then (its retry failed, or still going) takes them with it (doom). MediaFile plays a whole copy. Undo of a removed folder is RestoreFolder (POST /api/folders/{id}/restore, no body): RemoveFolder keeps the last ten folders each profile removed, with their downloads, in memory (goneFolders, not saved, gone at a restart), and the restore puts back the one with that id, for the profile that removed it; another profile's removals never evict it. The page sends nothing else, so no record of a download can come from it (as RestoreDownloads takes keys only).
  • History (history.go): a profile's plays, each its own sync record (named by when and what). Two devices that play one thing on one day each have a play of it; days shows the newest of a title or episode a day, and ForgetPlayed removes them all (Forgotten carries them, for RestoreHistory). A day is this device's: its zone (Core.local, the machine's; a test sets its own) tells a play's day and time, whatever zone the play was stamped in, so a play synced from another zone shows at its time here. Playing a title again replaces its play of the day History calls Today (record).
  • Notes (notes.go): a note's key is list|title|note, and a title's id may hold a |, so a thread is the keys whose part before the last | is the thread's key (Profile.thread), never a prefix of it. A note's id is made without one (newID). The key is also the sync record's name, so keys stay as they are. The first note on a title needs the title in the list as the list says (inList): a manual list's items, or a smart list's rules, tested on that one title (it must be in the profile's library, and its rules, with the fields had before the lock as Library has them, must match, or, while a provider the rules use does not answer (a field a plugin declares, unanswered), it must be in the list as it was (matching, which items shares: such a list keeps the titles it had, and their dates; a field no plugin declares matches nothing, as before); it writes nothing, and costs the same in a library of any size). So a smart list such as Backlog, whose items are none put there, takes notes too; shared or not, on every build. A title that is not in it gets ErrNotInList. A thread that has notes needs nothing more: the title's page shows it (titleNotes) and it takes replies, edits and deletes (hasNotes) whether or not the title is in the list now. A smart list's titles come and go by themselves (watching, a rating, a release window, a favourite, a provider's list), and a note is never to be left where nothing reads it. The cost: a title's page may show a thread of a list that the title has left, and the list's own page does not count it until the title is back. A first note on a smart list's title waits for the fields as opening the list does (answers are cached; a provider's list not asked yet is waited for, 3 s at most each); the page and a reply never wait. A shared smart list's book carries its rules and no items, so membership is each member's own (their library and watch state), and a first note needs the title in the member's own view of the list. The book carries the titles that have notes too (listBook), so a member reads any thread that exists; those who can edit reply (a follower reads only: ErrFollowed). Undo of a deleted note is RestoreNote (PUT, no body, the note's id in the path): EditNote keeps the last ten notes a profile deleted in memory (Profile.deleted, not saved and not synced, gone at a restart), and RestoreNote puts back the one with that id as it was (its id, text and time). It refuses an id with a | (validNoteID: a title's id may hold one, so the id x|N of title tt1 would otherwise read as note N of tt1|x), and an id not in that stash (ErrNoNote, so a note never written or another profile's cannot come back), and a followed list (ErrFollowed). It needs no membership test, since a deleted note is not a first note. Joining or leaving an account re-stamps the stash's authors as it does the notes (Profile.restamp).
  • Friends (friends.go, internal/together/friends.go, the migrations *_friends.sql and *_invite_peek.sql): an invite is an id and a secret, in its code and link. The service keeps the id, a proof (a hash of the secret) and the inviter's card sealed with the secret; it never has the secret. redeem answers the card and makes the friendship; peek answers the same card and changes nothing, so a link opened can say who made it before Accept (Core.LookAtInvite, GET /api/friends/invite/{code}, Remote.Peek). Both call invite_for, the one place the checks are (the profile is the caller's, the invite is there, unused, unexpired, with the right proof, not from the caller's own account, not blocked either way). No role can call invite_for, and peek is granted to authenticated alone. What refuses one refuses the other in the same words, a wrong proof being a missing invite. A service without peek (a self-hosted one not migrated) is answered together.ErrNoPeek, and the core answers nobody, so the page asks as it did before. Neither has a rate limit of its own: the invite's secret is 80 bits (the proof is derived from it), and an id alone redeems nothing. A look is one request. A profile's identity (the key pair its friends know it by) is made as a friend is (befriending, which syncs a profile this device has not synced yet first), never by the Account page (Friends only reads it, heldIdentity): a device that holds a profile's friends but not yet its identity lists each with since when and what the two share, and no digits to compare (the page has no Check button then), and marking one checked is refused (ErrNoDigits) rather than made against digits nobody saw. A block is carried out as it is made (Block, Unblock), and its record (blocked/) syncs; but a record can come or go without anyone pressing Block (a restore to a time it was there, the undo of one, a folder from elsewhere), so the end of a whole round has the service carry out what the records of the books it pulled say (tellBlocks, blocks_test.go): it blocks whom a profile blocks and the service does not (Client.Block takes "blocked already" for done), and unblocks whom the service blocks and the profile does not only when the book's sync state knows their record deleted (one it does not know, blocked a moment ago on another device, is left). It reads the account's blocks (Client.Blocks, one request) only when a profile has what this run has not had carried out (profileState.blocksTold, in memory): at a run's first whole round and as its blocks change, and never for a profile that blocks nobody and has unblocked nobody its state still knows of. A block of someone the service has no profile of makes no row (block_gone), and is made again by a later run.
  • The service's limits (internal/together, the migration *_account_limits.sql). A request carries or answers pageSize (1000) rows, the service's api.max_rows: Pull asks a page at a time from where the last ended, by (push, record) (seq=gte for the index, the or for the exact edge), so a push of any size is pulled; the reads of a table (all: members, lists, friends, guests, devices, profiles) ask page by page after the last row's key, and the usual one is a single request. A shared list too big for one push is rekeyed in pages of 500 records (Client.Rekey; builds before never pushed more at once, and one that pulls a page of 1000 rows of one push cannot go on): the first, rekey_list, replaces the key and adds its records, the rest are push_list's, each built from the list's last push (built), and only when all are in does rekey_done take the old records out (the new are named with the new key, so they sit beside them, and the list's cap is twice for every push to it while a rekey is open: lists.rekeying is set by the first page and cleared by rekey_done): a rekey that fails midway leaves the list as it was. Another push since (an editor's) is a 409, ErrChanged, and Core.Unshare reads the list again and starts over, as it does for a failed request (together.Transient, rekeyTries), so an edit is never replaced; and a rekey is remembered before its first page, on disk (profileState.Rekeyed, the list's last push before it, and every record of the list marked to be pushed), and forgotten only when it is done: whatever cuts it short (a failure for good, the app quitting), the owner's next round, after a restart too, pushes the records and then calls rekey_done, so no old record is left. That round is the device's that began the rekey, with her profile on it: taken off that device (or made local there) first, the profile's copy goes with what was marked, and nothing finishes the rekey. The old key's records stay at the service, and a copy made after has only what the new key carries (a list over one page, partial), while the copies made before keep the list whole. A copy made again does not take the mark: it lacks what only the old records hold, and would push the list without it and have them taken out. A pull that passes records over (the old key's: together.PassedOver) does not hold the round's pushes back; the round reports it at its end. A page and a push are held to bytes as well as records (pageBytes, pushBytes), under the service's 16 MiB. Row policies ask reached() and listed() once a query (profile in (select public.reached())), not owns() or role() of each row. A wake of another device has a channel of its own (account.woke) and starts a round after wakeSpacing (1 s) since the last began, and each wake within a minute of the one before doubles that, up to wakeSlowest (30 s); the wake is served after the wait, so none is dropped. A kick (what a person does here: an invite accepted, a party joined, a sign-in) is never held, and ends a wait. The news is one row an account, so it cannot tell an invite, a party or a friend from a push, and all wake alike. One round is several requests (the session, profiles, friends, each profile's seq and pull, shared lists, parties, the household), so a loop of changes by a friend or editor costs a device one round every 30 s at most, not one a change; it is the devices' cost that is bounded. The service's own writes (a news row and a Realtime event per change) are not, and the pushes that cause them have no rate limit of their own; a failing device's wake rounds are 30 s apart at the most frequent, and a kick still skips the failure back-off. Sign-ups: Supabase Auth asks before_signup before it makes an account for an address (so before it emails), which runs for new accounts alone: it allows one only for an email with a ticket (15 minutes, spent by the account; SendCode first calls ask_sign_up(email), which keeps it with its network and never refuses) and refuses the eleventh account of the ticket's network an hour, counted from rows that deleting an account leaves (limits.sign_ups; the network is the edge's own cf-connecting-ip as PostgREST passes it in request.headers, an IPv6 address's /64, or one shared bucket when there is none; calls from a network are serialised by an advisory lock; each is kept as a hash with a salt made again every hour, when every hash goes), so made-up addresses cannot use up the project's emails (auth.rate_limit.email_sent, a count of the whole project's). An existing account is never refused, and a second code for an address is Supabase's own once-a-minute limit. Housekeeping removes an account that has no device, profile, keyring or session after a day (purged.unused_after). Counted rows are capped (capped, after the row-level security check: 100 invites and 50 parties of a profile, 1000 blocks, 500 guests of a party; a push carries 16 MiB, and a profile keeps 100000 records, a list 50000).
  • Deleting the account waits (internal/together/devices.go, internal/core/devices.go, the migration *_delete_grace.sql). delete_account_later (Client.DeleteAccount) keeps the account in deletions with when it is due (purged.delete_after, 7 days; asked again, the first time stays) and writes the account's news row, so its devices listening hear of it at once. still_there is what a device asks as a whole round's first request (Client.Seen, in place of seen: no request is added): whether it is still one of an account's devices, and when its account is to be deleted. A service from before has no still_there (404, PGRST202), and Seen asks seen for an hour (Client.before, stillThereAgain), then still_there again (the service updated meanwhile), and at once once DeleteAccount has gone through. The core keeps it (account.session.deleting, Household.Deleting, the app told by news when it changes); while it is set a round pulls as ever and pushes nothing (Core.push answers together.ErrDeleting, and the round fails with it, in words: what changes here stays queued, stamped, as it does offline), so a device that was behind catches up, Put back of held deletions works, and the page and the top bar say so as for a failure. Meanwhile refuse_deleting, a statement trigger before writes to records, list records, profiles, lists, members, invites, parties and guests, refuses a device of the account with 423 (ErrDeleting by Error.Is; the service's words, with the time in UTC, are for a build from before, which says them as they are); deletions (unfriending, leaving, taking a profile or a device out) and devices coming and going are not refused, and a sign-in does not keep the account. keep_account (KeepAccount), by a keyed device as for deleting and a profile that manages the account in the core, takes the row out if its time has not come and its deletion has not begun, decided on the row as the keep takes it (began, clock_timestamp()), so a keep and a deletion at the same moment never both go through; after its time it is refused (410, ErrDeletingNow, in words; the core runs a round, which finds the account gone when it is), and a device of no account (its account deleted before it was told) gets P0002 (ErrRemoved, signed out here; delete_account_later too). ErrNotKeyed is a 403 in its words. When its time has come, its devices are no account's at once: still_there (and seen) answers not there, so a device is signed out and pulls nothing more of what is being emptied, and register refuses its email's sign-in (42501). delete_due deletes the account: a step takes 5000 rows of each of the tables an account fills (its profiles' records, its lists', and the history of both), and once those are gone the account as delete_account did (its users of no email of their own, then the account's user, which the rest follows), which ends the run, so what that took (its friends' news rows and memberships) is let go with the run's transaction. One caller at a time takes an account (its deletions row, for update skip locked). A step that fails is undone, said as a warning and kept on the row (failed, failure); the run passes that account by and goes on, and later runs take accounts whose last step went through first, so one account that always fails holds up no other. It runs from the service's own job (pg_cron, delete-due-accounts, every minute, one account and 5 s of steps a run, outside anyone's request; the job's own log is kept a day), so a quiet service deletes it too, and from still_there (and seen, a build from before's), a second of steps a request. docs/dev/service-ops.md (Deleting accounts: the job) has the query that says whether it runs and whether an account is late. Housekeeping (purge) is as it was. The migration takes the tables it changes first, in the order writers take them, and gives up after 5 s (lock_timeout) rather than hold them up. delete_account, which a build from before asks, is refused (426, words it says: update Mantle, nothing was deleted). The page (account/deleteAccount.js: Deleting, first on the page) says when, and keeps it for a profile that manages the account; it says who can elsewhere.
  • Who may replace the recovery sealing, delete a profile or delete the account (internal/together/account.go, internal/core/devices.go, the migration *_account_hardening.sql). A device signed in with the account's email and given no key is a device of the account that holds every profile (whole()), so it could do all three. keyrings.verifier is the SHA-256 of Key.secret() (a key derived from the account's), which prove (Client.Hold) compares with the secret a device shows, marking its row devices.keyed (a column no role writes). A device is keyed only when it made the keyring (a trigger: it has the key), was approved by a keyed device (a trigger on the pairing's sealed: the approving device's flag, not what the approved one says), or showed the secret of an account that has a verifier; prove writes a verifier only of a keyed device. Nothing else is evidence of the key: a device's card and the devices it approves are its own to make (so the migration keys no existing device: an account made before has no keyed device, no verifier, and the rule from before). keyed() is a whole device that is keyed, or any whole device while the account has no verifier. The keyring's update policy puts it in with check and a before delete trigger on profiles raises 42501 (not for a cascade, a level down: pg_trigger_depth), and delete_account_later and keep_account raise it, so a build from before is told an error and not a change of no row that it would take for done. A way for an existing account to gain a first keyed device without trusting a claim is a follow-up (no way found that does not trust one). Core.proveHere shows the key until the service answers that it is known: on a round at most every proveEvery, and every time before NewRecoveryCode, DeleteProfile, DeleteAccount and KeepAccount; a keyring just made is shown at once (NewKeyring). A service from before has no prove (Hold answers false, no error).
  • What a member of a shared list may write as another (internal/core/shared.go, the migration *_account_hardening.sql). Any member with the list's key can write its records, and a record's body says nothing true of who wrote it. list_records.writer is set by push_list as the profile whose member pushed it (the pusher names one it reaches that may edit, else its highest role; the owner may name anyone, and its rekey says who wrote each record as the pull answered (a removed member, or none: a row from before stays without, never made the owner's), kept in knownRecord.Writer; a row from before has none), Pull answers it (Record.Writer, never from the body) and Book.Writer is the profile a device pushes as. A note and a title are pushed sole (Record.Own, exclusive: the service cannot tell, a record's name is keyed-hashed): another member's push to the key of a sole row that has a writer is refused whole with 42501 unless the pusher is the owner. A list's members are the service's members rows (listMembers, on a whole round: named from the household's profiles and the friends, else "Someone"), and no member/ record is written or read (an older build's are passed over by applyListRecord, which leaves nothing for the round to answer with). A note is applied as its writer's (By, and the name, colour and icon of the list's member of that id), is not replaced or deleted by another writer, and the owner's rekey pushes each note named for its writer (recordWriter); a record with no writer is as it was. A title/ record is taken only from the owner, as the title it is named for, with no poster, backdrop or episode still, so only the owner's book has them (listBook; subject.pushes): an editor's would only be refused, and what a build from before queued of them is dropped, not pushed (pushRound).
  • A watch party is at its host's service. Its link names the site of its build's channel, and a release and a beta are at one service, so JoinPartyCode joins either's by the link of either. A link of another Mantle (a build between releases') is refused with ErrOtherMantle, naming its channel. Invitations by name are rows at the host's service. A code typed alone is this build's service's.
  • A watch party's code (internal/together/party.go) is its host's whole member id (80 bits of the hash of its party key, which is its channel) and the secret, 32 letters (readParty); its link ends in &v=2 (partyLinkVersion), which a build from before ignores. Hosts is a prefix test, so a code of 24 from a build from before (8 letters of the id, 40 bits) is joined as it was, and a build from before reads no code of 32 (its parser takes 24 and no other length, so a code cannot be longer and still be read by it: a party of this version needs this version, which the Invite panel and the guide say).
  • What is said on a watch party's channel (partymsg.go) is a message of one of ten kinds (hello, here, state, title, ask, queue, unqueue, chat, react, bye), each a type of its own beside the head every message has (partyHead: its kind, t, and the card of whoever says it). A message is read as its kind's type alone (partyHearers, each hearer made from its type, which says its kind: hearing), so each hearer has only what its kind says; one that does not read as its kind is not heard, and one of a kind this build does not know counts its sender and does nothing more. Each is written as the builds before wrote their one message type for all ten, the same keys in the same order and none when zero, so a party with a build from before on the other end goes on as it did (TestPartyMessagesAreSaidAsBefore, TestPartyHearsABuildFromBefore).
  • The sync id of a run (syncState.Device, newStampID in together.go): stamps each change the run makes, so two runs never tie, whatever data folder they started from (a restored backup, a clone, on the same computer or not). It is made at each start. Nothing depends on it staying the same: it is sealed in the records, is no registered device, and does not decide what is pushed. Stamps further ahead than a day (a clock that was wrong) are moved to the new id, to be set again (restamp). A record that comes with a stamp further ahead than a day of this device's clock (put right by the service's time: stampNow, below) is passed over, but not for good, since it may be this device's clock that is behind: the pull notes the push before the first of them and the earliest stamp (profileState.Held, HeldWall) and the log says that the clock may be wrong once, until nothing is held (clockSaid, in memory: a due pull that holds others again does not say it again). Nothing is pulled again until the clock is within a day of that stamp (heldDue: the clock set right, or time passing), and then the round pulls from the first of them again, once (what was applied is weighed as any record and skipped); a stamp that stays far ahead costs no request. What is held is not heard by the stamping clock (Heard), so a change this device makes while its clock is behind is stamped below the records it holds, and when its clock is right and they are applied, they win over a change it made meanwhile to the same record (it was pushed with the older stamp, and the others keep the later one by the stamps, so the devices agree; the change made on the behind clock is the one given up). It is not heard on purpose: the clock would then stamp this device's changes as far ahead as the records held, and the other devices would hold those, the way this one held theirs. Record.Seq is the push a record came in.

Profile PINs

Code: internal/core/pin.go, frontend/src/ui/pin.js, the routes in internal/server/profiles.go.

A profile can be locked with a PIN of 4 to 8 digits. It keeps a profile to itself in a household and is not encryption: anyone who can read the data folder can read every profile in it, and the core answers for the current profile whether or not its PIN has been given. The page asks for the PIN, and the core refuses only what changes whose profile is whose: SwitchProfile, ChangeProfile and RemoveProfile.

  • Profile.PIN is a random 16-byte salt and the PBKDF2 hash of the PIN with it (crypto/pbkdf2, SHA-256, 100000 rounds), saved with the profile in mantle.json. The PIN itself is not kept, and neither the salt nor the hash is sent to the page, put in a share or put in an export.
  • A profile asks for its PIN when it has one and is not the current profile with it already given. Core.unlocked is that one profile, in memory only: switching to a profile with its PIN, or setting a PIN on the current profile, opens it, and switching away, deleting it or restarting shuts it again. After a restart the current profile asks, so the page starts on the lock screen.
  • ProfileCard says locked (it has a PIN) and unlocked (it is the current profile, opened). The page asks for the PIN when locked and not unlocked (asks in ui/pin.js).
  • Wrong PINs are counted for each profile in memory, whichever call they came with. After the fifth wrong PIN in a row, each try waits 30 seconds, and twice as long after each wrong one, up to 5 minutes, even with the right PIN. A right PIN clears the count. The wait comes back as a WaitError.
  • The PIN comes in the request's body, so that no address holds it.
Route What it does
POST /api/profiles/{id}/switch {pin} Makes the profile current. The body may be left out for a profile that asks for no PIN.
POST /api/profiles/{id}/pin {pin} Counts the PIN as a try at the profile's, and answers true if it is, so a form can ask before it opens.
PATCH /api/profiles/{id} {name, color, icon, pin, lock} Changes the profile. pin is its PIN now, if it asks for one. lock is a new PIN, or "" to take the PIN off.
DELETE /api/profiles/{id} {pin} Deletes the profile, with the PIN if it asks for one.

A PIN not given or wrong is a 403, wrong too often a 429 with Retry-After (the seconds to wait), and a PIN that is not 4 to 8 digits a 400. A PIN's refusal is never a 401, which the page takes for the Android app's door and signs in again at.

Sharing a setup

Code: internal/core/setupshare.go, internal/server/routes.go.

A Share is JSON (mantleShare: 1, from, plugins, repositories, settings, lists, smart, titles). Its code is mantle1: and the JSON, compressed with DEFLATE, in base64url. ReadShare reads a code or a file's text, up to 4 MB once expanded, and answers ErrNotShare for anything else.

  • Setup is the current profile's whole setup. It takes no secret and no link that may hold a key. sharedPlugin leaves out a secret setting, an install link with a user, query or fragment (plain), and an address setting with such a link (shareable). addonShare turns a Stremio addon's link into the link at its site's root, if that is what it is, else into the site alone (Site). It also takes the addon SDK's JSON setup out of the path. Two lists of one name are told apart ("Name", "Name (2)"), so each can be picked.
  • A Pick is {plugins: [ids], settings: [group ids], lists: [names], smart: [names]}. SettingGroups are the groups of settings that can be picked (General, Playback, Appearance, Shortcuts), each with its keys and the line the picker shows. Share.Pick cuts a share to a pick. It keeps only the repositories of the plugins picked and the titles of the lists picked, and answers ErrBadSetting for an unknown group.
  • ViewShare is how a share would land in the current profile, to pick from again: each plugin's here (installed, install, configure or missing) and note in words, what it sets (the labels of its settings, an address with its host; for a plugin not here, the settings' ids), the repositories it would add, and which lists exist already. SetupView is the same for the current profile's own setup, with nothing to land.
  • ApplyShare brings a share in: the repositories of the plugins it installs, the plugins here (on or off, and their settings that are not secret; their addresses are kept while the plugin holds a key or a sign-in), the settings a group at a time (a group that fails is in Failed and the rest still come in; shortcut keys replace the profile's whole), then the lists the profile has no list of that name for. The plugins to install come back in ShareApplied.Install. The frontend installs each, asking for permissions, and applies the share again for that plugin alone.

The household's plugins

Code: internal/core/household.go, internal/plugin/, internal/together/.

What the household's profiles share is records of the household's profile at the account (household.go): plugin/<id> (how a plugin starts, what it was granted, its settings, where it came from), version/<id> (the version of a plugin installed from a link that a device installed or updated last, written only then, so its stamp orders the writes, a link gone back included; not for a private plugin of a profile kept on one device, nor yielded with a plugin removed and installed again), secret/<id>/<key>, signin/<id>, priority and managers. A device without a plugin installs it from its link as the version record says: a link that holds another has it held back (installSynced, HeldPlugin), until a device updates it, or the user installs what the link holds now (InstallHeld: what is allowed in its step is the household's, as grants are one record) or goes without it (DeclineHeld). A plugin the household has, installed by hand any other way, takes the household's grants.

A record's grants reach a plugin through Host.SetShared, as far as the plugin's manifest here asks (Regrant). A record says what the sender's user refused, and nothing else refuses: Plugin.Refused is a persisted mark per category (net, credentials, folders), set by Host.Grant with on false and by declining in the permission step (Core.declined), cleared by allowing, and never touched by Regrant, Replace or an update. Shared.Refused carries it, omitted when empty, so a record that refuses nothing is byte for byte what earlier builds wrote (a changed record is stamped and pushed as an edit, and one stamped by a device that was behind would win over what the household did meanwhile). SetShared applies a refusal only from the mark; in every other category it keeps what is granted here (keepUnlessRefused), bounded by this manifest as Regrant has it. So a device that cannot grant (a version asking for nothing, an update that names other hosts and waits for its user) refuses nothing for the others, and a record from an earlier build, with no marks, refuses nothing: a refusal made on such a build reaches a newer one when it is made again. The other way round is not reliable: marks live in the record, so an earlier build's record, which has none, replaces the marks of a newer build that refused (the record is newest-wins as a whole), and that build ends with its grant empty and no mark; a newer device that had not pulled yet keeps its network, so the refusal is lost for it until it is made again. Nothing cheap fixes this (an empty grant without marks cannot be told from an update waiting for its user), so a refusal is reliable only while every device has a build with marks. A record that allows the network names the hosts of the sender's manifest, which may spell them otherwise (an addon there installed from another link): a host granted here that it does not name stays, and when none is a host here the device stays as it is (failing closed), with one warning in the log when the allowance is new (it was refused here, and no longer is in the record). applyPlugin logs each permission a record allowed or refused here, saying it was on another device, as GrantPlugin logs the switch (only when it changes).

A profile's own lists

Code: internal/core/shared.go, listsold.go, sync.go, records.go, places.go.

A profile's own list is records, as a shared list is (shared.go), merged as every record is: the latest stamp wins. A list has own/<list>/name, own/<list>/rules (a smart list's; deleted when it has none), own/<list>/item/<title> for each title in it, and own/<list>/order (its titles in order). The notes on its titles (note/), each list's place among the lists (place/<list>) and the order of the lists for the builds before places (lists, and placed) are records of their own, as the library is.

  • A round finds what the user changed as it does for any record: a title added or taken out is a record set or deleted, a rename or a new order changes one record. Two devices that change one list apart keep both changes: each title, the name, the rules and the order are each the later of the two. The order of a list's titles is one record: the later order wins, and a title it does not name goes last.
  • The lists are in the order of their places (places.go, #354). Each list has a number, place/<list>, a record of its own (kept in the profile, Profile.Places), and the lists are shown lowest first, two at one place by their ids, the same on every device (sortLists, after every pull: settleLists). A list made goes after the others, at the next whole number (nextPlace). A list moved takes a place between its new neighbours, and the others keep theirs: a reorder keeps the most lists whose places already rise in the new order, and places the others between them (placeLists); one moved after the last stays below the next whole number, where a list another device makes meanwhile goes. So lists made on one device and a move made on another, at about the same time, are both kept whichever is later, and so are two lists moved on two devices; a list moved on two devices goes where the later move put it, between its neighbours as that device had them; lists made on two devices at once come at one number, by their ids. Places so close that no number is left between them are spread out again, those of that whole number alone (spread): from the whole number up to halfway to the next, not further, where a list moved after them on another device meanwhile goes, so it stays after them. There is always room for that (moves to the end crowd lists just below the next whole number, so the first of them has none to spare); all are given again, 1 to n, only if the places do not rise after it. A place that comes for a list not here yet (a shared list's, pulled before keepShared brings the list) is kept for it (addList), and is no record of this device's until the list is here (placeRecords); when the list arrives, its place is known as it came, at the pull's stamp (keepShared), not taken for a move made here then, which would undo a move she made since on her other device; one kept for a list that is gone is dropped as the state is read (loadPlaces). A list deleted takes its place with it (dropList), and the place's deletion goes up with the list's; one a restore brings back under a new id (renewDeleted) has its place, and its id in placed, renamed with it. Requests: a move is one record for each list moved (the one order was the whole list of ids), and a list made or deleted has one more.
  • The builds before places go by the order of the lists as one record (lists), and a place is of a kind they do not know, which they leave alone: never taken for deleted, never pushed. This build writes lists with the order its places make, and the same under a name they do not know (placed), as the order changes, so they show what it does. A lists pulled with another value than the placed known with it was written by one of them, a reorder made there: the lists take its order as lists was always applied (the lists it does not name go last: first those this device learned from elsewhere, by their ids, the same on every device; then the lists made here that no push has carried yet, in the order they were made), and the places follow it (settleLists). The places it moves take its stamp, not that of the device that found it, so a move made later on another device, stamped and not pushed yet, is not undone by a device that merely took it (#353's rule for the order of the lists). While such a build is in the household, what it does to the order is one record still: its order (pushed as it reorders, or as the set of lists it has changes) wins as a whole over the moves made before it, and the lists it makes go after the others in its order. A state is read in the order it was saved in: one from before places has none, and is placed so (1 to n, the same on devices that agree on the order); one such a build saved since (it keeps the places as it read them) has the lists it moved placed again, the others keeping theirs (loadPlaces). What a load gives is no move of the person's: the first look knows those places, and placed, as set at no time (yieldGiven), so the places the household's other devices have win over them, and those the account lacks go up as early (stamped at that look, a device updated after another one had moved lists won over the moves with the order its files had, and the household ended in one nobody made). But where the order of the lists waits to go up (lists queued: a reorder such a build made there and did not send), that order is the person's, and is weighed after the first pull (settleLists): when no later order came, the lists are put in it and the places follow it at its stamp, as for an order such a build sent, so only the lists it moves take that stamp (yielded and no more, the order went up as the account's had it; stamped at the first look, before the pull, it undid a later move made on an updated device). Going back to such a build loses nothing: it shows the order lists has, and keeps the places in the state as it read them.
  • A list exists as its name does. setOwnList makes a list when its name arrives (the name is ranked first in a batch, ownRank, so the records after it find the list) and takes it out when its name is deleted; a title, rules or order record for a list this profile does not have, or has shared, is passed over (and not known as applied, so it comes in when the list does). Delete wins. A list deleted is gone on every device with every record of it at the service (its own records, the notes on its titles and their read marks: listRecord), whatever another device changed in it before it heard of the deletion, whatever the stamps say. The device that deletes it finds its name and every record of it known here absent, and the round pushes all their deletions (absentOwn: the name's absence also notes the list in profileState.Deleted, dropList; a list that is here with no name among the profile's own records is shared, its records are its book's, and is not deleted). A shared list that is the profile's own again (left, or no longer shared: ownAgain) is no more in Deleted. A device that pulls the name's deletion takes the list out whenever it has one (a restore that brought a deleted list back gives way to the account's deletion, as the restore says; where it has none the deletion is a no-op, and Deleted is never what decides it). A list that is shared here is not an own list, whatever the profile's own records of it at the service say, and none of this is done for it (subject.shared): they are those of a list left on another device, and it may be the user's own again within the round (keepShared, which runs after the profile's pull, and ownAgain), so the deletion that goes with it would otherwise be queued for a list that was kept (with its notes), notes it, and queues the deletion of every record of it that it knows, each at a new stamp (applyPulled), so what it changed meanwhile goes too; deletions of the list's other records that come in the same pull (the name is ranked first) are taken as the account's own, and pushed back by no one. A record of a deleted list that comes in later, from a device that had not heard of the deletion, is not applied: its deletion is queued at a new stamp. No stamp is compared to decide this, so a split push (a list over 500 records is two pushes) or a pull between its parts cannot leave a list half kept. Deleted is a list of ids in together.json, oldest first and at most maxDeleted (1000; older ones are forgotten, so a device offline across more than 1000 later deletions can bring a list back with a late record of it); a build before this one leaves the field as it is (keep:"-"). A list made again has a new id and is not touched. A list that is shared again under the id it had (left, then added again) keeps its own records live at the service while it is shared (nothing deletes them: the sharing is its owner's act, and another device of the profile that has not heard of it still reads them as its own list). What a member's devices show of it is the book alone: a member's first round with a list takes the book's records for everything, and starts from no titles (keepShared), so a title the profile added while the list was hers, or one a device brought since builds from the own records, is not in the shared list, as the owner's book does not have it (by design: a title added to the list while it was hers and not yet synced is dropped from it on the share too; it stays in the library, and at the account as an own record). So does every copy a device has no sync state for, its owner's too: on each device of hers but the one that shared the list, the copy made there when the list first comes (empty: her profile brought there, or there already when she shared the list, which took its id at the service then); and on any device, one there with no state for it (her profile made local and joined again, restored from a backup on a wiped device, what was synced lost). Its name and order would otherwise go up as changes made there, newer than the list's, on every member's device. A copy made here drops a state kept for an earlier one (a profile taken off the device and brought back: removing a profile leaves its states). The device that shares the list makes its state at the share, saved with it (ShareList), so that copy is the list's word and its first round pushes it as it is (a round of what changed here alone that comes before the share's whole round leaves the list shared, with that state: it changes no sharing, below). A profile made local keeps its shared lists as its own (unlink); joined again to the person at the account (JoinProfile, one more request: the memberships), each that is a shared list the person is still in is that list again before anything is merged, so its records do not go up as the person's own (under the list's id they stayed at the account, and came back in the person's list once it was no longer shared, on a device that had not heard of that). The clear and the marking of the book's records as known are under the lock that marks the list shared, so a title added meanwhile is neither lost nor taken for the book's. Those own records stay at the account, and a device that has not heard of the sharing still shows them in its own list. A list left and then deleted on one device, while another device of the profile still has it shared and hears of the leaving only after it has pulled the deletion, goes there too (leftDeleted, in keepShared): the latest record of its name is that deletion, the profile's last doing to the list as its own, so the list is not made the profile's own again there (it was, and its name went up newer than the deletion, and the list came back on both: #339). The device that leaves stamps the list's records at the leave (StopSharing makes the look then), so a deletion after it, before any look, pushes the deletion of its name too, and the other devices find it; deleting a shared list (DeleteList, which leaves it first) knows its name alone, as set at the leave, so its deletion is the one record that goes up (the list's titles were never the profile's at the service). That device takes out the list and its notes (the book's), not the profile's own records of it (its sort, place and read marks): the deleting device's push carries their deletions, in the pull before it, and a deletion of them made there would be later than a sort made on a third device while the list was shared again. A list shared again after the deletion, and left again, is kept: a device that sees it shared again, in the rows a whole round reads after its pulls, forgets the deletion, which is older than the sharing, and a later leave makes the list the profile's own as before (TestLeaveDeleteReshareLeaveKeepsList); one that did not see it, and has its round between that leave and the leaving device's push, takes the list for deleted, but does not note it among the deleted lists (Deleted, whose records are never applied), so the name the leaving device pushes brings it back there (TestListKeptByALeaveAnotherDeviceMissedComesBack). In that last case, what that device had changed in the shared list and not pushed yet is not in the list that comes back. The same holds when what ends the sharing again is its owner (it stops sharing the list, or takes her out), not a leave of hers: then no device of hers pushes the list's name of itself, and a device of hers that missed the sharing has the list gone until one that saw it has a whole round (it makes the list hers then, and its name goes up), or for good if none does. A title in a list is in the library. The library's record of a title (added/<id>) and a manual list's record of it (own/<list>/item/<id>) are separate records, so after a pull that applied either, a title in a manual list that is not in the library is settled by the later of the two stamps (settleLibrary, after applyPulled): a removal later than the list's add takes it out of that list, as RemoveFromLibrary does by hand (an item with no known record was put there here just now, and is later than any removal), and an add later than the removal puts it back in the library; what it changes goes up in the same round. Smart and shared lists are not touched (a shared list's titles are its members'). The same for a put made here: a title the library already has, put in a manual list on a device that has not heard another's removal of it, does not change its library record, so the removal would be the only stamp on it and win (the title gone from every list). So when a look stamps a manual list's item record, it stamps the library's record of that title again (restampPuts, in changedHere), its value as it was (so the date added and the order stay), as an add at the put: the later of it and the removal wins, as in settleLibrary. A removal the add beats is not applied, so settleLibrary puts nothing back for it and nothing is pushed twice (what it does put back, a title a list's later add keeps against a pulled removal, it pushes in the same round). A title new to the library has its record stamped once, as new. Requests, here: one more record in the push for each title put in a list that the library already had (not one for a title that is new to it). Requests: a list's deletion is one push of the records the deleting device knows (batches of 500 as every push), once; other devices that hear of it push only what they know that the push did not carry.
  • A deletion is remembered for 90 days (tombstoneAge, pushRound), as every record's is.
  • What is not a list's: when its titles joined a smart list is each device's own and in no record.

Upgrading from the betas before 0.3.0 (listsold.go)

Betas before 0.3.0 (beta.1 to beta.7) wrote a list as one record, list/<id>: the list whole (id, name, items, rules) and, from the build that merged lists, beside it crdt: the stamp of each title coming in or going out, of the name, the rules and the order, and the stamp the list was last written whole at (the floor). Each device kept the same stamps in the lists field of together.json. The old merge engine is gone; what is kept of it is the reading of those stamps as records, which is all the latest-stamp rule needs.

  • The device's own state. A saved List is as it was, so an own list loads unchanged. A device's first round with a profile (upgradeLists, before the round looks for changes) turns what it knew of each list into what it knows of its records, each at the stamp it was set at and not pushed yet: from the stamps in together.json, else, for a list unchanged since the old record it last knew, from that record at its stamp (a whole write: every record as of then). What the user changed since the device last synced is found by the round as any change is, and stamped now. A list changed since, by a build that kept no stamps, has none to know: all of it is as of now, except that the titles of the record it last knew are out unless another device added them after (Floors), so what the user took out stays out. The old state and the old record's mark are dropped.
  • The account's records. A list/<id> that comes in (an account's first pull, or a device that has not upgraded writing it) is read as the records it is (readOldLists, oldRecords): each at the stamp it carries, a whole write all at the stamp of its record, a deletion as the list's name deleted (a stamp with nothing in it, or further ahead than a day, counts for nothing, the whole write's floor too). An old stamp has no device, so one is made from the value's hash: two values set at one stamp are read in one order on every device. Each is weighed like any record pulled, so a device that upgrades later merges what it kept in the old form with what the first wrote, once, by the stamps; reading the same old record again changes nothing. An old deletion is the exception to the weighing, as delete wins: it is read for a list among those deleted here too when the name is known as set (a state put back from before the deletion has the list again, and gives way to it), and passed over when the name is known as deleted (a deletion made here, later, is not replaced by it). Where the name was set after it (a rename on a device that had not heard of the deletion), it is read as of now and goes up as the name's deletion, so the account's latest of the name is a deletion for a device that reads the two at once as for one that read them in turn. A whole write leaves out the titles that came in before it and it does not name: titles known here older than it, and records pulled older than it, are deleted as of it and go up as deletions (Floors, kept in together.json).
  • The old records stay. An upgraded device never writes list/<id>, and deletes one only when a build before writes it again for a list deleted (readOldLists: not read, its deletion queued): a build before reads it, and would take its deletion for the list's. So a build before goes on seeing the lists as they were when it was last written, writes its own changes to it (which upgraded devices read), and never sees a change made on a newer build: it does not crash (a record of a kind it does not know is left alone) and loses nothing. An upgraded device writes about as many records as it has titles, once, and one or two for each change after.
  • What differs from the old merge is only how two concurrent changes that the old engine settled by what they changed are settled: by the stamps (and their devices, or the value's hash for old stamps). A floor is dropped with its list's name. A title either device has is in the list unless the other took it out after, so nothing is lost. A title two devices add in one tick goes last, in no fixed order between them, where the old merge put it by its stamp. A device that pulls records an upgraded device pushed first, identical to its own queued ones, does not push them again.

Telling what friends and shared lists did

Code: internal/core/activity.go, devicename.go, losing.go, friends.go and shared.go (where they are noted), frontend/src/app/activity.js, frontend/src/app/pages/account/ (lately.js, checkFriend.js, deleteAccount.js).

Something another person did reaches a device only as a change of what the rounds read: a friend's answer to an invite (keepFriends), the friendships there now, and the shared lists the profile is in (keepShared). Where a round finds one it notes it (Core.note) in syncState.Activity: friended, unfriended, shared and unshared, each with its number (Last), the profile here it happened to, the other person's name and the list's, kept for a week and at most 50. Nothing is asked of the service for it: the request counts of a round are as they were. Each is told to its profile once (syncState.Told, by profile), as the page says a profile's happenings up to a number are told (POST /api/activity/told, naming the profile Activity showed it: the core refuses with a conflict, ErrProfileChanged, when the profile in use is another by then), and Activity answers the profile watching its own alone. The page tells one batch at a time; news that comes during one is told when it ends.

  • Each device tells for itself. Last, Told, Looked and ListShare.New are in a device's own sync state and lists, not in what syncs. Two devices of a household that are running when something happens each find it in their own round and each say it; opening a list on one leaves its dot on the other. Syncing them would not stop the second device saying it: both rounds find it within moments of each other, and the mark would reach the second only after it had told, unless a device waited for the others, which is polling the design does not do. What a person sees is the toast on the screen in front of them, so one on each is not wrong, and the dot goes when the list is opened where it is seen.

  • What counts as new. A profile tells of what arrives after it has looked at its lists once (profileState.Looked), so a device just signed in, or a build that begins to tell, finds its lists and says nothing of them. A list shared by a profile of the account (the household's own doing, by ps, the account's profiles, whether that profile is on this device or on another) is not told, nor is its ceasing to be shared. A list the profile left or stopped sharing itself (StopSharing) is made its own at once and is not found by a round, so it is not told either. A list new to the profile has ListShare.New until OpenedList (the page opens it), and its chip shows new.

  • What it cannot tell. A friendship ended and a list left on another device of the household look to a round as the other person's doing, so they read "You and Sam aren’t friends any more" and "… isn’t shared with you any more", which is true of both.

  • A friend keeps Since (when they became friends, from the invite) and Checked, the safety digits the person compared and found the same: a friend is checked while their digits are those (FriendCard.Checked). A friend unfriended and befriended again is a new Friend (a new Since, not checked) whatever the digits. Both are in the friend's record, so they sync to the household's devices, and a build before them reads the card without them. An invite says when it runs out (together.InviteLasts, the service's seven days).

  • A device's name. A device is named once it holds the account's key, by its own name: its host name, MANTLE_DEVICE_NAME (the Android app gives the phone's model), or in a browser "Firefox on Windows" read from its user agent (browserName). A name already on its card stays (a person's, or the name an approving device gave), so the card is read first (one request, once for each device, or again if the name it calls itself changes: syncState.Called), and the write is made only over the card as it was read (NameDeviceIf filters the update on it): a name given between the read and the write is kept, and the device looks again next round. A device asking for the key has no key to seal a name with, and the service sees what it says as it asks: its kind alone, which is why an ask says when it was made (its end less askLasts) and with which email, and the approving device may name it.

  • What deleting takes. Losing counts the friendships and shared lists of the profiles here (one profile's, or all), from the core alone. The page adds the profiles and devices it already has. A profile not on this device is not known, and the counts say "at least". Deleting the account is confirmed in the request as well as in the page: POST /api/household/delete carries typed, the email (or delete on a device with none) as the person typed it, and Core.DeleteAccount refuses with ErrNotConfirmed (400) without it, before anything is asked of the service. The page and the core are one build in the desktop app, the web build and the Android app, so an older desktop app has an older core, which does not check and ignores typed: it asks delete_account after its page's own confirmation, which the service now refuses (it would delete at once: see Deleting the account waits), saying to update Mantle. Only an older page left open against a newer core (a web tab kept across an update) sends none, and is refused until it is reloaded.

What sync costs the service

Code: internal/core/sync.go, news.go, records.go, internal/together/live.go, and the tests synccost_test.go, syncpace_test.go.

The service is paid for by its requests and its rows, so a round (syncRound, sync.go) is of two kinds, and keepSyncing looks every 3 seconds (lookEvery, with no request) for which is due (due).

  • A whole round makes the checks: Seen, the account's profiles (each with its last push), friends and the invites answered, the lists it is in and their last pushes, invitations, and the asks (some eight requests), then pulls what the checks say changed and pushes what changed here. News (a kick) asks for one; without it one comes every 5 minutes while Realtime is connected (syncSlow, account.loop.live, set from Listen's joined; a reconnect runs one at once, for what came meanwhile) and every 30 seconds while it is not (syncEvery). SyncNow and the actions that kick (a friend, a list's member, a joined profile) make one too.
  • The look stamps. Each look, before it decides on a round, stamps what changed here (stampLocked, which due does, and the retry wait does while rounds fail): the records of every synced profile, shared list and the household that differ from what was synced (changedHere, as a round does, with the lists from before this build turned into records first, upgradeLists), with no request; a round stamps as it begins too (syncRound), so a change made while the loop waited on news (up to wakeSlowest), or whose round fails at its first request, is stamped. A change then takes its stamp a few seconds after the user made it, not in the first round that gets through: a device started with no network has no round for hours, and its change, stamped when the network came back, would beat what the user did meanwhile on another device (#319). Not covered: a change made while a round is out on a request (up to roundEnds) is stamped at its subject's start in that round, or at the next look. A stamp is kept with the sync state, after the state, as a round keeps it (account.save: a start before the round does not stamp it again, later), and once: a look that a round follows leaves the write to the round (account.file.unsaved; round writes what is left when it ends, and the loop writes at a look that no round follows), so a change is written once, not at the look and again by its round. A paced record (progress) is stamped when the pace says it is due: syncPace after it was last pushed and after it was last stamped (account.loop.pacedAt), so with no push going up (offline, the service down) a title that plays is stamped and written once every 90 seconds, not every look; a book that has still to give way is left to its round. The look is cheap when nothing moved. A look makes every record of every synced profile (5,000 titles are some 20,000 records: 20 ms and 15 MB on a desktop, 90 ms in the web worker), so the profiles' and lists' are skipped while the state marshals to what it did at the last look: marshal hashes what it marshals (the tick of Persist asks each second) and bumps account.loop.rev when that differs, as do KeepHere and a plugin removed (what keptHere reads and the state lacks). unchanged skips them while rev is the last look's, is not 0 (no state marshalled yet, as in a test with no Persist: every look is made), the last look was less than lookWhole (30 s) ago, and no paced record the last look left for its pace is due (look.heldUntil). account.flush (a playback ended, PushNow) moves rev too, so the look it asks for is made whatever the last tick saw: a change made since the tick, and a record held for its pace, go up as Mantle quits or the page goes out of sight. The household's records are made at every look: what they hold (a key) is not all in the state, and they are a handful. A record of a kind written once (recordKind.once: titles) that is already known is not marshalled (recordsOf: it is there with no value, as a look asks only that it is), and what due asks after the look is a scan of what is queued (queued), not another build. An idle look costs 0.2 ms (0.9 ms in the worker); a change costs one look, 20 ms (90 ms), a few seconds after it. The stamps go by the service's time (#320). Two changes made on devices with no pull between are ordered by their stamps, so with the devices' own clocks minutes apart a later removal or rename would lose to an earlier change on another device. Each answer of the service carries its time (Date, in whole seconds), and together.Offset keeps how far it is ahead of this device's clock (Offset.Timed wraps the Together client's requests: UseTogether). An answer puts the offset between two bounds, the Date's second and the trip apart; the answers that fit narrow them, so a slow answer costs nothing. One that does not fit is kept aside, and the offset in use stays, until the next: if that one agrees with it, the clocks moved (one was set, or drifted while nothing was asked) and the bounds start again from the two, else it was an odd answer and is forgotten. Only an answer that answers or refuses the request (2xx, 4xx) is read: not a redirect, which may be a network's sign-in page, nor a gateway's failure. The offset in use is their middle, moved only by a second or more (so the answers' jitter does not write the sync state again), none under a second (a clock that is right stamps as before), and held within a day (a service whose clock is wrong cannot move the stamps further). A clock set while the offset stands (by the person, or the network's time) would stamp by the offset of the clock before. Each reading of the clock (Core.reading, together.Reading) carries the monotonic clock beside it (the one time.Now's times carry; a test's clock carries none and is its own, unless it gives one: Core.mono), and when the two have moved apart by a second or more since the offset was last borne out (an answer that fits it, or the start), the clock was set: the stamps go by the clock alone, as before the offset, until two answers agree on another. The monotonic clock is to count the device's sleeps, or a sleep would look the same: Windows' does (Go's is the interrupt time there); on Linux and Android Go's is CLOCK_MONOTONIC, which stops in a sleep, and what the device slept since the process began is added from CLOCK_BOOTTIME (slept, sleep_linux.go; a test stands in for it with Core.sleptHere). Where it may stop for a time that is not known (the browser's performance.now(), which some systems stop in a sleep, and macOS), a clock that went on further than it is taken for a sleep and the offset stands, and only one that went back is taken for set (Reading.Pauses, sleep_other.go): there, a clock set forward goes by the offset of before until two answers agree. An answer whose trip on the two clocks differs (the clock set during it) is not read. Core.stampNow is the clock put right so (Offset.Service): the stamping clock (together.Clock) goes by it, and so does whatever weighs a stamp against now (what is held as further ahead than a day, restamp, the deletions' horizon, an old list's floor). It is kept in the sync state (syncState.Offset, in ms, as saveNow finds it), so a start with no network stamps by the last one; a build from before keeps the field as one it has no place for, and stamps by its clock alone. There is one offset, the service's the device syncs at. The web build reads Date where a page may: PostgREST's answers (every request of a round) expose it (Access-Control-Expose-Headers), Supabase Auth's do not. What it does not close: no stamp goes back (Clock: one is after every one given or heard), so what a device stamped by a clock that was off with no offset to put it right (with a build from before, before its first answers, by a clock set while Mantle was not running, which the saved offset does not know of, or by one that drifted while it was offline) holds the stamps after it until the time catches up, and with a clock ahead its changes win meanwhile, as before. Pulling before stamping would not close more (an offline device cannot pull, and a stamp made after a pull is later than what the user did meanwhile elsewhere) and would cost a request at each change. A device that heard another's change still stamps its own after it (Clock.Heard).
  • A round of what changed here alone starts when changed says a profile's records, or a shared list's (listBook, unless it is followed), differ from what was synced (the look stamps them, and queued asks what waits to go up, the household's too; PushNow asks the same; a shared list is looked at without its title records, which walk the title cache: an owner's title record that changes by itself waits for the next change or a whole round), and pushes (a request) with the profiles and lists as the last whole round found them (account.seen): a profile it has not heard of is not unlinked, a household profile is not made, and nothing is pulled, since no news came (or the next whole round pulls it). Nor does it change which lists are shared here (keepShared): it syncs those shared here that the rows have and that have had their first round, and makes no list shared or the profile's own again by them, nor gives a copy its first round (a copy a join just marked shared: the rows may seal a key a rekey since replaced, and none of its records would open), since a list shared, stopped, left or joined here since is newer than those rows (going by them, a list just shared was made its owner's own again, its state gone, and then cleared to the book's empty word; one just stopped or left was marked shared again, and cleared the same way).
  • Echo. A push makes news for the account, which its own device hears too, and news says nothing of what changed. News that comes within echoWait of this device's own push costs one request (newsCalls: the profiles with their last pushes) and starts a whole round only when some profile's last push is not the one this device already has, which its own push advanced (push), or the account has a profile this device has no state of, or this device has one the account lacks; so another device's push is never taken for the echo. News later than that is a whole round (it may be a friend, a list or an invitation). Within the window, a friend, list or invitation change is not seen by that check: it waits for the next whole round, at most 5 minutes (closing it would take a kind on the news row, or a request for the lists' seqs too). A news event is account.loop.news, apart from kick, which actions here make. The connection is live while Listen is joined; its heartbeat ends it when nothing came on it for two and a half beats (a half-open socket after sleep), and the whole rounds are every 30 seconds again until it joins.
  • Playing. A paced kind of record (recordKind.paced: progress) does not start a round of what changed here alone more often than syncPace (90 seconds, since the last push), and is not stamped more often (the look, above). PushNow has it go up at once, within a time (a round in flight is not waited for beyond it; one call at a time, and it checks again for a change once it has the lock): as Mantle quits (stop), and when the player says the page is going out of sight (the out flag of POST /api/play/{session}/at, sent on pagehide and visibilitychange: a tab put away, a phone locked, Android's app left). When a playback ends, account.flush has the next look push. Not covered: a process frozen or killed with no such event, and sleep, which leave the last 90 seconds of progress unpushed; the profile has it and saves it, and a round stamps what differs from what was synced when the process runs again. A play's history record is named for its time (as every build has named it, so builds from before read it), and a playback keeps the time its play entered the history (Playback.began), so a film is one record, not one every few seconds; a play of the same title that day, after it, replaces it once.
  • Deletions older than tombstoneAge (the lists' rule) are dropped from syncState.Known (pushRound). The service keeps its sealed rows, which it cannot tell from other records.
  • Measured (TestSyncCostOfPlayingAFilm, TestSyncCostOfAnIdleHour; two devices of one profile, one playing and one idle, Realtime connected): two hours of a film were 480 pushes, 480 history records at the account and some 12,500 requests, and are now about 100 pushes, one history record and about 1,400 requests; an idle hour was 1,920 requests and is 192.

The account's parts and their locks. account (together.go) is a struct for each of its parts: session (the service and this device's session there), keys (keyring: what opens the account's profiles here), pair (pairing: asking another device for the key, and the asks answered here), state with file (syncFile: together.json and how its writing goes), plugins (housePlugins: the household's plugins as they come here), loop (syncLoop: what wakes the sync loop and paces it), rounds, parties and the restores. account.mu guards every field of them but those whose type locks itself (the atomics, sync.Map, rounds, parties, pluginHold, together.Offset, and together.Clock for its stamps) and those set as the account is made; the struct groups its fields so, rather than a comment on each saying which lock is its. Lock order: account.round, then account.mu, then the core's mu; a watch party's own lock (party.mu) comes after the account's and before the core's, but for one place that holds the core's and only tries the party's (inUse, titles.go: TryLock, giving up while the party is busy, so it never waits the wrong way round). The locks of the parts that lock themselves are taken last, with nothing taken under them. A round holds account.mu only between its requests, never across one, and no request is made holding the core's mu.

Locks in a round. c.round runs a round holding account.round (one round at a time, ending after roundEnds, a minute) and account.mu, which guards the account's state. The round's requests (requests, sync.go) let mu go while they are out, so the Settings account page, Household() and sign-out never wait on the service for a round. An answer that comes back after the session or the account changed (account.session.epoch) is dropped and the round ended; a round ended so is not counted as done. Every call that changes the account's state (profile links, lists, asks, devices, the household, keys, friends, parties: account.change, or account.round alone in the friend calls, which ask the service without mu) and newsCalls take account.round first, so they wait for a round in flight instead of changing what it has read. Registering the device and making the account's first key stay under mu throughout (once per device; their answers make secrets).

A service restored from a backup (#360; what the maintainer does is in The Together service).

  • The cursor. A device keeps, for each book (a profile's, a shared list's, the household's), the last push it has seen (profileState.Seq), and pulls what came after it. The service numbers a book's pushes up from the last it gave (profiles.seq, lists.seq) and never back. So a number below the device's means a service whose database was put back to an earlier time. From then on the service numbers what any device pushes from the backup's number, and a device that was ahead would pull none of it until the numbers passed its own.

  • Where it is found (foundBehind). Only where the service has just told the number:

    • the profiles a whole round reads (the household's among them);
    • the lists' last pushes keepShared reads in a whole round;
    • a push's answer (a push numbered no later than what the device has seen).

    Not the profiles as the last whole round found them, which a round of what changed here alone goes by. Its own push has moved Seq past them since, as has another device's push between its pull and its push, so that is no restore. It follows that a device sees the restore only while the service's number for the book is below its own. Ahead of the backup by n pushes, it no longer sees it once n pushes of that book are made since, and the first device to find the restore makes one as it sends back what was lost.

  • The mark. The book is marked (profileState.Behind, kept with the sync state, so a round cut short or a restart reads it whole still, even once the service's numbers have passed this device's). It is read whole at its next round, from the first push (pullRound), and what it holds is applied by the stamps as any pull is. The cursor is set to where the service is only once the read is whole. So Seq stays as it was meanwhile: a build before this one, which has no place for the mark, goes on as it did (no worse), and the next whole round finds the service behind again if the mark was lost.

  • What goes back up (pushBack). After a whole read, what this device has that the service lacks goes back up:

    • each record whose value here is the one it knows as synced, at its stamp (written after the backup's time, it is on the devices alone, and a device signed in afterwards would never have it);
    • each deletion it knows of a record the service has nothing of.

    Not what is kept here (keptHere), nor what the brake holds, nor a newer version's record (subject.foreign: this version has no value of it, and its device sends it again when it finds the service behind). Nor, of a list, a record another member wrote: the service would take this member for its writer, so each member's devices push their own, and a follower's none. What a device knows of a record's writer is the service's: a push keeps the writer each record went up as (an owner sends another member's as theirs), and so does a rekey, so the owner's are its own still, and a member's note stays the member's to change. A member that is not the owner never sends a note another member wrote (pushesRecord), not even one it has later than the service, as a restored service has it older: the service would refuse it, and with it the whole push, every round. That note is its writer's devices' to send again. A record made and deleted after the backup's time comes back from a device that had not heard of the deletion, unless the device that deleted it finds the restore too and sends the deletion again, by the stamps.

  • The brake (holdAgain, bringBack). A deletion this device had synced, of a record the service has from before it, goes up again by the stamps, as in any pull (actRepush). When there are as many as the brake holds (massLoss, the rule of holdLost), they are held on a profile instead, since a service put back to before a mass deletion may have been put back for that. Each is held as the service has it (Braked) and noted (Again), and Settings asks as for any held deletions. Each device that had the deletion holds it, and the first answer goes for the household:

    • Delete them everywhere deletes them again, stamped now, and a device that holds them applies that and lets go.
    • Bring them back forgets them and reads the book again. What it held then comes back as a restore brings records back (bringBack, as putBack does), and goes up. It is stamped as the person pressed (profileState.PutBack, heard by the clock first, as it may be a run's before this one, by a clock that was ahead), not when the round runs (which may be hours later, offline): later than the deletion each device holding it has, which takes it and lets go, and earlier than what anyone changed after the press, which wins. An own list deleted here comes back under a new id (renewDeleted), with every record of it the read found, the held and the rest: what a device that still showed the list did to it after the restore (a title added, a note, its sort) goes with it, at its own stamp when that is after the press. Under the old id each is deleted again. It is a new id, not the old one taken out of Deleted, because a deleted list stays deleted under its id: every device that pulled its deletion, a build before this one among them, deletes it again whatever comes. While the hold lasts, what another device writes to the list is not deleted again here (delete wins waits), so it is there to go with the list.

    A list's book and the household's have no brake (as for holdLost).

  • A rekey cut short (Rekeyed) is finished by taking out of the list what was pushed up to the push it began after (RekeyDone), which must be the old key's records and nothing else. A restored service may have the list from before the rekey, from between two of its attempts, or from after, and may have had pushes since under numbers up to it, so no number from before tells. The whole read does. Below the first record it opened, none opens with the list's key as the service has it now, so that is the cut. First the list's records all go up again under that key (queued, as the rekey sends them, each as its writer's), so that what was only under an older key is not lost; what goes up next is numbered past the cut. A read that opened nothing does not finish it. RekeyDone waits while the book is still to be read whole (pushRound).

  • Not covered.

    • A profile joined, or a list first shared, after the backup's time is not in the restored account, and each device unlinks it (unlink, ownAgain), keeping its records here.
    • A list brought back takes what the read found of it, so a change made to the old list after that, on a device that still shows it and has not heard of the move, is lost. When that device pulls the old name's deletion, delete wins: it takes out every record of the old list whatever its stamp, and the new id has none of the change. Restore synced data (renewDeleted) has the same window. Closing it needs the move to say where the list went, so that such a device moves its own records instead.
    • A device that does not sync while the service is below its number, and a build before this one, miss what was pushed under the numbers they had passed, until it is changed again (#384: the service is to say it was restored).
  • Cost. Nothing more while the service is not behind: the numbers are those a whole round reads, and the pushes answer. Once it is found, it costs a read of the book whole (a request a thousand records) and a push of what it lacked, per book, per device; and a push of what is brought back, once, by the device that answers.

Record history, and the service's brake on mass deletion

Code: supabase/migrations/20261113000000_record_history.sql (and 20261114000000_history_at_walk.sql, how history_at reads, 20261114100000_history_budget.sql, a version a day, and 20261115000000_history_writers.sql, who wrote a list's versions), internal/together/history.go, internal/core/syncrestore.go (and syncrestore_changes.go, what a restore changes), sync.go (pushRound, push), and the tests internal/together/history_test.go (the local stack) and internal/core/syncrestore_test.go, syncrestore_more_test.go (the fake service, which models both).

History. The service keeps only each record's latest version, so a push gone wrong was final everywhere. A trigger on records and list_records (keep_record, keep_list_record) now keeps what a write replaces, in records_history and list_records_history: the sealed body as it was, its push (seq), when it was written (written, a column of the live row too, set by the trigger) and replaced, and for a list's the writer. A deletion is a replacement (a tombstone), so a deleted record is kept as it was. Nothing is plaintext: the bodies are sealed on the devices, and the stamp that orders a record against its other versions is inside the body. The tables belong to nobody a device can read: they are read through history_at, a profile's by a device that writes it (writes), a list's by its owner, and for the last 30 days only; a profile or list deleted takes its history (the foreign key). A hard delete (a rekey's fresh and rekey_done, which drop the records of a key that is gone) is not kept: the old-key bodies could not be opened by anyone.

Reading it. history_at answers a page (1000 records at most, after a key) by walking the book's history in the order of its index (records_history_of: book, key, replaced), from a record to the next that has a version since the time, one look in the index a record (and one more for a record written after the time: its earliest version, to know whether it existed then), until the page is full: a page costs the records it passes, not the size of the history, and never reads another account's rows (measured on a service of 3.7 million rows: a restore of a profile of 100,000 records with 560,000 versions, 3 days back, in 1.2 to 1.3 s, its pages in 10 to 25 ms). A look must be that and nothing else, since a plan a connection keeps is taken for every step: one made from a wrong guess (the history empty when the plan was made, or taken for empty just after a mass deletion filled it, or a time a few minutes back) read the table at each step, or the prune's index of every account's versions by time (records_history_old). So the function turns off whole-table reads, bitmap scans and sorts (enable_seqscan, enable_bitmapscan, enable_sort; jit too, which such a cost would set off), and each look asks for its rows in the order of their keys, which only records_history_of gives without a sort: the look for a record's earliest version takes its key as a range of one (key >= and <=), since with key = the planner drops the key from the order and the prune's index gives the rest. TestHistoryAtOfABigProfile checks what it answers at size against the rule; supabase/checks/history_at_stale.sql checks, with the statistics of a history taken for empty, that no page reads a table whole or by the prune's index (both on this computer's service, in mk test).

A record keeps three rows an hour at most. A deletion replaced by a deletion keeps nothing (no value is lost by it). Any other replacement is kept in one of three rows of the record's hour (purged.history_every, from the first replacement in it): the first is a row of its own, the version that began the hour, and the last two are two rows (rolling) of which each later replacement in the hour takes the place of the older, with the version it replaces and the time of the replacement. So what a loop of values and deletions keeps does not grow with its pushes (measured: 30 records of 200,000 characters, ten cycles of deletions and values: 600 rows and 76 MB for a 7.8 MB book before, 60 rows and 7.8 MB with two rows an hour, and 15 MB after five more pushes of deletions, which was the bound: twice the book; with three rows it is three times the book, 23 MB), a film's position, written every 90 seconds (syncPace), would be three rows an hour while it plays (it is one a day, as a record written often: below), and a bad write (a push of garbage, or of deletions, the case of #208) leaves the version just before it, exactly, whatever came before in the hour: it is one of the last two rows. One more write of the record after the bad one (another device's, an edit of a deleted record, a restore's own push) does not push it out, so a second restore to the same time reads what the first did; a second write after it does, and then what is left of before is the hour's first. What it costs the restore: for a time inside an hour in which a record was rewritten, history_at answers the version before the hour's latest writes, which may be newer than the time asked (the first kept after the time, for a record known to have existed), never an older one. An interval of 0 makes every version a row of its own (a test uses it).

A version a day. Two kinds of record keep one version a day, not three an hour. A record written often: replaced more than purged.history_often times (5) in its hour (from its first replacement kept a row of its own, which begins the hour). Watch progress is written every 90 s while something plays, so every play of nine minutes or more is caught, and its history, of little use to a restore, was most of the history's volume. The service goes by how often it is replaced alone, which it sees already (when each version was written), and knows nothing more of what a record is. What the device pushes is unchanged. The record keeps the version that began its hour, the record as it was before its hour's writes; its hour's rolling rows go, and for a day from that first replacement nothing more is kept. The live record counts its replacements in the hour (rewrites, set by the trigger, 1 when a new hour begins). It and the budget's columns below (daily, kept) are the service's: a device is granted the records' columns it reads by name (records: profile, key, seq, body, gone, written; list_records: those and writer, sole), and not these. And every record of a book over its budget (a profile, or a shared list): one that keeps purged.history_budget rows (20,000, about 13 MB) or more, until pruning brings it under, so that a client rewriting many records in a loop (a sync bug flipping them between devices, or an account doing it on purpose) keeps a row a record a day at most, not 72. Such a record, replaced within a day of its last version that was kept a row of its own, keeps nothing more, and has no rolling rows. A version replaced after that day is a row of its own again, which begins the next. Nothing is refused, and the live record is written by every push as before (TestSyncProgressGoesUpEachPaceAndIsKeptADay: the core's pace and what it pushes are unchanged). push decides the budget once, before it writes: its records are written daily (a column of the live row) if the book is over (history_over: the book's counts of the days of history_for summed, 31 rows at most, in their own index). keep_record reads daily, and says on the row when it kept a version (kept); push counts those as it writes (its returning) and adds them to the book's day once (history_kept, list_history_kept: a row a book a day). Nothing counts down: the days pruning takes leave the sum with their rows, and prune_history deletes their counts too. So the budget costs a push two statements, whatever its size, and a record nothing (measured: below). The count is an estimate within a push (one that crosses the line still keeps by the hour) and a day (a row is counted on the day it was kept, and pruned by when it was last replaced; the rolling rows a record written often lets go stay counted for their day). What a restore gets of a record kept a version a day: history_at is unchanged, the first version kept since the time, so for a time before the record's first change of a day it is exact; for a later time it is the record as a later day's first change found it, newer than asked; and it is none, so the record stays as it is, when the record changed only within the day of the version kept before the time. It is never an older version: one from before the time would undo what was changed before the time the person picked, which the restore promises to keep ("What's new since then stays"). What it costs, accepted for the bound. Every replacement in the hour counts toward the five, a bad write and the writes after it among them, so #345's promise that the version just before a bad write is kept, through one more write, holds only while the record's replacements in that hour, the bad write and those after it included, come to five at most: four edits and a bad fifth keep it, three edits, a bad fourth and one more write keep it, five edits and a bad sixth do not (the restore then answers nothing for the record, which stays as the bad write left it). A burst of more than five replacements in an hour (six titles added one by one replace the library's order six times) keeps that hour's first version and nothing more for a day: a bad write later that day keeps nothing, so a restore to just before it answers nothing for that record, and a restore to before the burst answers the hour's first version, without the burst's edits. A book over its budget does the same for every record: the day's first version replaced, and nothing more until a day after it.

Pruning. What is older than purged.history_for (30 days) goes in prune_history, which runs after the upsert of any push (it is a statement trigger, which fires for an insert ... on conflict do update whether or not it replaces a record), at most once an hour (history_pruned, a row of its own): no job, and no device asks. It never holds up or fails the push it is in: it takes the pace's row only if nobody has it (for update skip locked, so another writer passes by and no account waits on another's push), deletes at most 5000 rows of each table a push (the hour is over only when a batch was not full, so a backlog is worked off a batch a push), and catches an error in it, a statement timeout or a cancel too (query_canceled: it runs in the push's statement and spends its timeout), with a warning in the service's log: the push goes on and it is tried again with the next.

Cost. It adds no request: the cost is in the rows. A row is the sealed body (the record's JSON, 28 bytes longer, in base64), a key of 24 characters and some 300 bytes of heap and indexes. Measured on records the core makes (a library of 150 titles with a 380-character overview, posters and four keywords, 30 of them series of 40 episodes; a list, marks, progress and history; sealed length in characters): a mark, a library entry, a progress or a history record is 150 to 220; the library's order is 2,500 for 150 titles (about 17 a title: 35,000 for 2,000); a title's record (title/, a snapshot of what it is) is 5,700 on average and up to 23,000 for a series of 40 episodes (more for a longer one); a list's name and items are 200 to 800. So what weighs is a title's record, when it is replaced, and the library's order, each time it changes. A typical account (300 titles, a few adds a day, an hour and a half of playing a day) keeps about 0.5 MB of history a month (the library's order three times a day at about 5 KB, progress a row a day of each title played for nine minutes or more (three an hour of play before #376), a hundred marks and plays of 0.5 KB); its live data is about 2 MB. A heavy one (four profiles of 3,000 titles) keeps about 5 MB of the library's order a profile a month, and one bulk change of every record (what a bad push or a restore does) keeps each record rewritten: about 70 MB for the 12,000 titles, and a little more for the rest. The worst case, one profile at the 100,000-record cap rewritten in full, is about 270 MB for 30 days if 40,000 of its records are titles of that size. A thousand typical accounts are 0.5 GB; a hundred heavy ones, with a bulk change a month, about 9 GB. Supabase Pro includes 8 GB, and extra is $0.125 a GB a month. A loop of rewrites had no bound but the record's (72 rows a day): at the cap, about 145 GB in 30 days. With the budget, a book keeps about 20,000 rows and then a row a record a day: at the cap, rewritten all day, about 3 million rows (2 GB) in 30 days. What it costs a push of 1,000 records that replaces them all, measured on this computer's service (a history of 300,000 rows of 400 profiles; main's functions and these in turns on the same data, medians of 15): keep_record alone, 31, 40 and 46 ms before as each record's version begins an hour, takes one of the hour's last two or replaces one in place, and 33, 37 and 46 ms now; 14 ms for a record written often or of a book over its budget, which keeps nothing in the day (37 and 41 ms before), and 25 ms as a record turns written often and lets its rolling rows go (44 before). The whole push (public.push, as a device calls it): 44, 50 and 58 ms before, 44, 55 and 63 ms now (within the runs' spread), and 24 to 25 ms for records kept a version a day. history_every, history_for, history_often and history_budget are settings of the one purged row; changing any needs no deploy.

The brake. push and push_list refuse (428, code PT428, together.ErrMassDeletion) a push that deletes wipes(n, live) of a book's live records: 250 or more, or 100 or more and more than the book has left (the device's own brake for what a lost state would delete has the same rule), unless it says it was meant. The service cannot read a record, so it cannot tell a tombstone from a value: a device says which records of a push are deletions (gone, set for a record with no value) and which it meant (meant), in the rows it sends (recordRow; a service from before ignores both). A deletion is counted only if the record is live at the service (records.gone, kept by each push), so a deletion repeated counts nothing. A rekey is not a deletion. A build from before sends neither, so its pushes are not braked: nothing it sends can be told apart. That is the limit of the brake: it holds against this build and its successors, and a bug in an older one is caught by history alone.

The brake takes the book's row first (the seq bump, then the count), so two devices pushing at once cannot each stay under the line and cross it together: the second counts what the first left. A refusal rolls the bump back. What the book has live is what is not a deletion (gone), which a build before makes too many: it cannot say its deletions are, and they stay in the book. So a push of this build says, on each deletion it did not mean, how many records of the book it holds live (recordRow.Live, from Book.Live, set by pushRound from profileState.live), and the least of that and the service's count is the book's. (A service from before ignores the key, and a push that does not say is counted by the service.)

On the device (sync.go), what is meant is what a person did while Mantle ran, whatever its size (a 600-title list, the whole history). The deletion brake (holdLost, above) holds what a saved state lost at the start, most of a profile's records, and asks the user (Braked, ConfirmHeld, PutBackHeld). What it leaves, the service's brake backstops: as the files are read at the start (checkFiles, lookAtStart, whether or not anyone is signed in; the records of each book are built once for it and the brake, readOnce) what the sync state has as synced and the records lack is noted (profileState.lost, not kept), whether it is a profile's, a shared list's book or the household's. It is the start's, not what a person did after it with Mantle running (a signed-out run, or the seconds before the first tick, included): when changedHere takes such a record for deleted, it is not meant. It is marked on the record (knownRecord.Unmeant, kept with the sync state, so a failed push or a restart does not make it meant), and is cleared when the record is pushed, or made again or given way (giveWay); What ConfirmHeld lets go is never in lost (the brake held it), so it is meant, even before the run's first round. A push of a profile sends those apart and last (pushRound) once there are massDeletion (100) of them, so the service refuses them alone, and an extra request is made only then. When it refuses, the book gives way to the account at once (takeBack, from the records just read, so what the person changes after is theirs): what the account has comes back, a list found missing is not a deleted one (it is taken out of Deleted), nothing is deleted anywhere, and the error is logged and kept as a notice (massRefused; rounds.done takes it out of the round's errors, so the round is judged by the other books' errors alone, and keeps it as a Refusal apart from trouble, detail and failing, under its own lock: the items, when, and together.Words' plain wording for a 428; two books' refusals in one round add up). Household.Refused says it until a round goes through without one or refusalShown (10 minutes); a failure of the same or a later round has its own words and never loses them to the notice. The service's status stays in the log ("the account refused to delete 312 items that looked lost, and they are back (the service answered 428 PT428)"). The round is not failing by it: the service answered and the records are back. The Sync group says Household.Refused in its words while the core says it (sync-refused, account/syncStatus.js), apart from trouble, which it reads only when failing. The count sent with a deletion not meant (Book.Live) is taken as each push goes, so the second push of a round counts what the first left. The other change of the round was pushed before it. A person who meant them deletes them again with Mantle running, which is meant. It is not retried, so it is not asked again every round: Transient is false for it, and it is gone with the give-way.

A restore (Core.RestoreSynced, POST /api/household/restore/{of} with {"at": "<RFC 3339>"}, api.restoreSynced; of is a profile's id here, household, or list: and a shared list's id; the page is Settings, Account, Restore synced data, account/restore.js) runs a round, reads the versions the profile's records had at that time (the page's time is by this device's clock, and is put in the service's time first, as the stamps are now, in one place for the restore and its preview: serviceTime; Household says the time as it was asked) (Client.HistoryAt, which opens them with the profile's key), applies those that change the record here as if pulled but stamped now (applyPulled: what a device applies is decided by the stamp in the body, and an old one would lose to the very version it should replace, so the service cannot restore on its own), queues them and pushes: every device pulls them like any change. It puts back each record that was replaced or deleted since that time; a record that did not exist at that time is kept, so a restore deletes only what was a deletion then. It answers how it went (Restored): how many records it changed here (records: a version the same as the record here, of a record this build does not make, or kept here by hand is not put back, sent or counted: changing; the records a book answers for are its subject's knows: a profile's and the household's are those this build makes by name, knowsRecord and knowsShared, and a shared list's those of listKind it pushes, so a newer build's record is left as the account has it, as a round leaves it), how many of the account's versions did not open and so did not (unopened: PassedOver, sealed with the key before a rekey), and whether what came back is still to reach the account (unsent: the push after it failed, or was cut short; the records are here and queued, and the next rounds send them, as any change made here, and the log says why). An error means nothing came back here. It saves (account.save, both files of one moment) once the records are applied and queued, before anything goes up, and a push of several batches is one moment saved as it begins (push, above): a device stopped during the push (killed in the background, a crash) starts again with what it put back here and queued, and its next round sends what had not gone, rather than starting from before the restore, or giving way to the account with the first batch alone (the profile part-restored on every device). While it runs, Household.Restoring says so, by the profile's id here with the time it goes back to, and the pages are told as it starts and after it ends (news), so another tab, or the page opened again, shows it running and asks nothing more. A device that is signed out is told so (together.ErrSignedOut) before its key is looked at (signing out forgets the key, but that is not why). It keeps what is new since the time (a record with no version then), and the page says so ("What’s new since then stays, and so do friendships": a friendship is the service's, not a record); it counts what came back as changes (records). The page asks it with the profile's name typed, offers a day of the last 30 and a time on the half hour (an hour ago at first, never one to come), shows the row's restore running from Household.Restoring, and says what came of it there; what came of it is kept for the app's run (lastRuns), so a page left meanwhile says it in a toast, with the date, and shows it when opened again. Its preview (Core.PreviewRestore, the same route with "dry": true, api.previewRestore) runs the restore's first round and its history read, at the same moment (serviceTime), and weighs the versions against the records here as the restore does (changing, changesOf, syncrestore_changes.go): what would come back, by kind, with a few names each (Changes: titles back to the library or a list, or the household's plugins; titles gone again, as they were not there then; own lists back, made again; and every other change, counted by record, named by the list or title it is of; a title's details and the orders are not counted). Nothing is applied. What it read is kept by the book and the time asked (restores.previews, in memory, for 15 minutes; a preview of another time does not take its place), with the book's last push as its round found it; and the restore of the same book and time in the same session takes it (takePreview, which forgets every preview of the book) and reads the moment it read, though the offset moved since. It puts back those versions: a history thinned in between (#376) cannot make it do less than the preview said. If the book was pushed to since the preview's round (another device, the one that went wrong perhaps, still pushing; the book's last push is asked after each round, a profile's by profilesAt and a shared list's by ListSeqs again, so a push during the round is counted), the restore reads the history again and adds the versions of the records the preview had none of, which changed since (withVersions: a version at a time is only lost from the history, never another, so the preview's stand). With nothing pushed since, it reads nothing more. The restore answers its own Changes, by the same rule, so the page's words after are its own. The page shows the preview in the question and offers Restore only once it has come and found something to change; a question closed ends its preview's request (AbortController), and a preview whose request ended keeps nothing. Its timer knows it by its number, so what it read is let go once it is replaced or taken, not 15 minutes later. An undo (Core.UndoRestore, POST /api/household/restore/{of}/undo, api.undoRestore) is offered for 10 minutes after a restore that reached the account (Household.Undo, until when; Restored.Undo), in the app's run and session. It reads no history: the history need not keep what the restore replaced (#382 keeps nothing on a push for a record written often, or of a book over its budget), and this device had it as it restored. So the restore keeps, in memory for those 10 minutes, each record it changed with the stamp it gave it and the value and writer it replaced (undoOf.records, replaced: at most the book's records as they were; a restore after a mass deletion replaced deletions, which hold nothing). The undo runs one round (the account's latest comes here), stamps what changed here meanwhile (changedHere), and puts back what still has the stamp the restore gave it, as it was here just before the restore, stamped now: anything changed after the restore, on this device or another, stays (Restored.Left). A record whose deletion the brake held when the restore ran (Braked: the files here had lost it, and the account had it still) is put back as the account had it: what the restore replaced there is the version itself, if the account had the same (then the restore changed it here alone, and the undo counts it as put back, with nothing to send), or else the account's value, which this device knows the hash of and not the value, read by the restore before it pushes (heldAtAccount: a pull of the book from its first push, made only when the restore changes such a record at the account to a value, which is after a loss and rare; a version that is a deletion changes nothing, as the record is not here). A value read is kept only if it has the hash this device knows, and a value that did not open (PassedOver) is not read. A record whose value was not read so is held (replaced.held): the undo leaves it as the restore put it, counted in Left, and sends nothing of it. So a held deletion is never made a meant one, nor an older version put back, and a value another device wrote there is not lost. What the undo would put back is let go when it may no longer be undone (keepUndo's timer, which knows it by its number; Household lets go of one no longer offered as it reads them) and when the session ends (endSession, where a sign-out, a sign-in or another service bumps epoch, drops what restores kept: what a preview read too), so a household's restore does not hold its plugins' keys for longer than its Undo. A record the restore made (it had no value here: a deleted list's records under the new id) is deleted again. A list the restore brought back that anyone changed since (a record of it the restore did not make, a title added or a note; or one not as it left it) stays whole, as it is, and all the restore's records of it are left: deleting its name would take the list with every record of it, the change among them. A list nobody changed goes with every record of it: its name's deletion goes first and drops the list here (applyPulled, which takes the deletions of its other records that come with it for the account's, as a pull's are), so putBack counts and pushes those too, and nothing of it is left live at the account. What it put back and what it left add up to the restore's records. Both the restore's and the undo's second step (putBack) pull the book first (pullRound, one request, empty unless something was pushed meanwhile; the household's plugins it changes are asked anew, as those the restore changes are: touched is made before it; a restore then reads what the brake held, heldAtAccount, above), and then check the session and the book again (restoreHere), as the requests let the account go: a sign-out meanwhile, during the read too, stops it before anything is put back here (ErrSignedOut). What was pushed while the history was read, or during the first round after its pull, is here before what is put back is weighed against what is here (changing), so a change another device made then is restored too, and one made since the restore is left by an undo. If the account can't be reached, it is weighed against what is here, and what comes back goes up with the next rounds (unsent); but a restore that must read what the account has of a record the brake holds fails then, with nothing put back. An undo is not undone in turn, and a restore that did not reach the account (unsent) offers none. A restore or an undo of a book that runs already is refused in words that name it (ErrRestoring: that profile, that list, what the household shares). A shared list is restored by its owner: history_at(shared := true) lets the list's owner alone read its history (an editor, who could not write every member's records either, is refused 42501), and the core asks it of a list a profile here owns (ErrNotListOwner otherwise), with the list's key and book as the round found them. history_at answers who wrote each version of a list's record (20261115000000_history_writers.sql), so a member's note is pushed back as theirs (Record.Writer, which the owner may name): it shows as theirs on every member's device, and they may still change it. The owner's device knows it as theirs after the push (push keeps each record's writer as it went up), so a later rekey of the list (a member taken out) writes it again as theirs; an undo puts back each record with the writer it replaced. What the household shares (household: its plugins, their settings, keys and sign-ins, their priority, and the managers) is restored by a device that holds every profile (managing: the service's writes() refuses a device of one profile too), and the plugins it changes are asked anew (refreshTouched).

A list that was deleted is in the Deleted of the device that deleted it and of each that pulled its deletion as an own list's (a record of one is never applied there: delete wins, and a rename that raced with the deletion must not bring it back), so it cannot come back under its id; a device that took it out as a list left and deleted elsewhere while it was shared there (leftDeleted) does not note it, and knows the deletion of its name. renewDeleted gives such a list a new id (deletedList): the records of a deleted list that the history has, under their new names (own/<id>/..., its notes and read marks, its sort, its place in the lists order), are restored as a list made again, which every device takes like any new list; the old id stays deleted. A list renewed here is kept by its old id (profileState.Renewed), so a second restore here to the same time puts its records back under the id it came back under, while that list is not deleted here itself (deletedList), and makes no second copy; a restore from another device makes its own. One restore of a profile runs at a time on a device: a second is refused (ErrRestoring, 409), and a restore from another device at the same time is not stopped. A restore takes two short rounds (restoreBook, putBack) and holds nothing between: the history of a book of 100,000 records is a hundred pages, asked for within restoreEnds (10 minutes) with no lock, and each of the two rounds runs under that limit too (roundFor; a round has a minute), so Settings, and whatever waits for a round (change), are not held for it. The second round checks that the session is the one it was asked in (epoch) and that the profile is still here and synced to the same book, and applies and pushes (a push cut short by the round's limit goes up with the next rounds). Only a profile that manages the account may do it (manages), as it rewinds a person's whole profile, a list or the household's records on every device; a device of one profile alone restores that profile and the lists it owns, and no other.

Questions to providers

Code: internal/core/network.go, news.go, answers.go, playback.go, playsource.go, provided.go, frontend/src/app/ (useLoad).

Everything a plugin supplies is a question: a list's page, a search, a title's details, a title's sources. Questions are cached, and each is asked once however many pages wait for it (network.go).

  • An answer has a lifetime. Lists live for 30 minutes, details for 6 hours, sources for 10 minutes.
  • An old answer still serves while it is asked again. If the new one says something else, the pages hear of it (news.go) and ask again quietly, keeping what they show.
  • Some answers are kept on disk between runs (answers.go): compressed files, up to 64 MB, the least recently used going first. Nothing that says something about an account's secrets is kept. Sources are not kept, since a source's link can carry an indexer's key. A download does not save a private source's link either (a server's key in it), nor a finished one's direct link: it keeps the source's id, and finds the source again by it when it needs a link (refind).
  • A failed answer with nothing before it is forgotten after a while, so asking again tries again.
  • A call can say what it has so far (mantle.so_far), held on its answer while it is on its way. Search shows it: an addon's catalogs that have answered, while a slow one has not.
  • A question can be in a lane of its plugin's: a newer one in the lane stops the older if it is still on its way, and forgets it. Each plugin's searches are one lane, so the queries typed past never hold its Lua states (four a plugin).

Pages that fill in

A provider can be slow. So an API page answers at once with what the providers have given so far, and names what is still pending. Asked again with ?wait=1, it answers once the next answer has come, or all have, or the request ends. The frontend's useLoad does this: the page shows what it has and asks again until nothing is pending. /api/news tells open pages when to ask again.

Playback

playback.go runs the steps of getting a source ready, and the player shows each. A playback has a stage:

Stage What it means
prep Finding sources and trying one.
pick The profile must choose a source (none found, or all failed, or Picker is on).
transfer A resolver is preparing the release. The core asks its progress every 3 seconds.
play A stream is ready. The player has its address.

The core fails over to the next ranked source when one fails (bestLeft ranks what is left; a download that cannot start tries the next as well, fetchDownload, before any of it has come). Told a source did not play, it asks the stream's host (whyFailed): a 404 or 410 is said as "the provider no longer has it", since a picture reports a file that is gone as it does a format it cannot play. The player tells the core once for each stream that is on screen (useStream: gave is cleared as the render that has another stream comes, not in an effect after it, since the new stream's picture can tell the core before such an effect runs, and be told again by its next error; so one picked again after it failed is told again). It can prepare another source while one plays and switch to it. The player reports where it stopped, and the core records progress and history.

One thing plays at a time. Play ends the playback still going (another window's, another browser's) as if it were closed. The player that followed it gets a 404 from its next request (ErrNoPlayback), closes, and says "Stopped: something is playing elsewhere".

How finders, resolvers and the check for cached releases combine is in provided.go. A resolver declares what it accepts. Both resolvers that come with Mantle accept "torrent" and nothing else. A finder's direct link (no info hash) skips the resolvers and is a source with no resolver, whose where reads "Direct", counted as cached. A release a resolver's check lists as refused is not offered through that resolver.

The container: repackaged streams

Code: internal/container/, frontend/src/app/video.js, stalledStream.js.

Reads MP4 and Matroska files in parts: keyframes, length, chapters. Repackages one as HLS (fragmented MP4, each audio track its own rendition) for a player that cannot switch its audio tracks itself; one whose keyframes are too far apart for a browser to hold, its sound alone, its picture relayed as it is, both from the same reads (TooFar, Picture). A stretch between two keyframes of more than 16 MB and at least two seconds is cut in parts, which start without a keyframe: their playlist titles them part, and leaves out #EXT-X-INDEPENDENT-SEGMENTS where there are any, and for a file that begins mid-GOP, whose first segment starts without one too. hls.js 1.7 does not step back from such a segment: its fragmented-MP4 remuxer takes a keyframe index left undefined, not -1, for none, so it counts the segment as independent and the browser drops its pictures, and it has no setting or hook for that. Parts cannot go (a stretch is up to 96 MB, and a segment is read and remuxed whole), so the player goes to the stretch's keyframe itself, when it is sought into a part, resumed in one, or hls.js is started again in one (video.js). Started from the keyframe, hls.js goes there itself once its first part is in, and on to where what is loaded begins when that is a little after (by less than its skip tolerance: the sound's first part can begin a few ms after the keyframe); that seek is the player's own, not a viewer's seek elsewhere, which would let the picture go (ownSeek); a viewer's seek (seekTo) never is, even one that lands where the player's own would. It is the first where the WebView is late to say it has its metadata, as on a busy runner (Resuming in a part, the WebView late to say it can). That is the workaround for hls.js's independent = keyFrameIndex !== -1 in its pass-through remuxer (keyFrameIndex is undefined for none), issue #51; the report to make upstream is in PR #99's description.

A stream's host that asks to be left alone (429, or 503) is no stream that is broken. refusal (internal/core/hostbusy.go) turns its answer into a BusyError (ErrBusy, the wait it asked for in Retry-After, 0 if it did not say), which every reader of a stream (Core.ranged: the head and index, the parts, the frames, the probe SourceFailed makes) passes on. replyOr answers it 429 with Retry-After, the one row with a PIN's wait (ErrWait); the core's repackaging and keyframe reads do not log it as a failure, and ask again on the next request. The wait itself is held per host by polite.go (a host that asked is answered busy at once, without being asked, for as long as it asked), so the other sources on that host are not tried meanwhile. SourceFailed asks the stream's host for a byte, as it does for a file that is gone (404, 410), and one that answers busy is no failure: nothing fails over, no source is marked failed, and the answer is that 429. The player (frontend/src/app/video/hostBusy.js) waits and asks the same thing again: hls.js's loader is patient (a 429 or 503 is retried after Retry-After, else 2, 4, 8 s up to a minute, whatever the policy says of other errors), the ask to repackage and the failure report go through patiently, and a plain stream is loaded again after the wait. hostBusy is what the player says (BusyNote: "The server is busy (it asked Mantle to slow down). Trying again in 8 s."); a minute of refusals is over, which the note says, offering the next source, and which stops the waiting: nothing then plays the stream as it is.

Each stretch of a file is asked of its host once. A stretch is read once for the picture and every sound (Media.recent, two stretches at their largest), and the segment of each sound asked for is cut from the picture's read as it comes (cutSounds, cuts.go: sounds are small, a 4K picture's stretches are not), so a sound that trails its picture beyond what recent holds does not read the stretch again. A picture played apart is the browser's own, which loads a minute or more of a long film and, after a seek, asks for it again (it holds less than it says is loaded): its chunks are kept by bytes, as many as fit in apartKept, the least recently asked for let go of first, not two stretches (tens of seconds of a 4K film). apartKept is sized to what the browser can ask for again, about the 150 MB of picture it keeps, and to the platform (keptOn): 256 MB on a computer, 128 MB on Android (a twenty-fourth of the phone's memory if that is less: 85 MB with 2 GB, at least 48), 48 MB in the web build, whose WebAssembly memory is never given back (a seek further back than that asks the host again). It is memory held while the film plays: a chunk let go of while it is still being read for nobody is given up (its read ends), Close lets go of all, and the web build and Android collect at a half more than what is live, not the default double (debug.SetGCPercent(50), start in cmd/mantle/app.go; a soft memory limit would do nothing there, as the heap does not come near it: measured with 128 MB live and 1 MB reads, the heap peaks at 283 MB with a half against 375 MB, and with 48 MB live at 179 MB against 231 MB, twice the collections, 2 to 4% of a core even when reads come at 20 GB/s) so the garbage of the reads is bounded (TestApartCloseLetsGoOfWhatIsHeld, TestApartGivesUpAChunkLetGoOfWhileFilling, TestApartKeptByPlatform); and since even that is less than a long film, its sounds are cut from each chunk of it as the chunk comes (cutApart, from part, for the segments that end in it, held bytes only: no host read), for every sound asked for, and what a sound read itself is kept (samples): a seek anywhere in what the picture has loaded, forward or back, asks the host for nothing (TestSeekWithinLoadedAsksNoHost, TestSoundBehindItsPictureAsksNoHost). The cuts are bounded as before (cutKept); a sound is forgotten when the sound asked for last is cutRecent segments from it, whichever way (TestApartForgetsASwitchedSound, TestApartForgetsASwitchedSound/after_a_seek_back: a seek back asks an earlier segment). A segment whose chunks have not all come (the later of two read at once may come first) is tried again as each chunk comes, while it ends in what is held, and one being cut is not cut again by another chunk's goroutine (TestApartCutsWhatCameOutOfOrder); one that lacked a chunk that is not held (before the window's edge, or a sound stored apart from its picture) is tried again when that chunk is held, not as each comes (cut.missing, TestApartDoesNotRetryWhatCannotBeCut); one that lacked a chunk which came while it was being cut (the chunk's own pass skipped it as busy) is cut again at once (TestApartCutsWhatLackedAChunkThatCameMeanwhile), and a cut that ends after Close notes nothing (TestApartCutAfterCloseNotesNothing); nothing is looked through while no sound is asked for. A stretch longer than recent holds cannot be cut whole, and its sound is read from the host as it was. Only the picture reads ahead (aheadOf). A read that fails or is given up before all of it came is not joined, but what came of it is kept and the next read of the same stretch goes on from there. The pictures of the seek bar (Core.Keyframe) are read one at a time, no faster than frameEvery, and kept (framesKept); frames.js asks no more than the core's pace and none while the stream's host is refusing. The sound beside a picture played apart is asked for up to twice as far ahead as at 1×, however fast the picture plays (REFILL_SPEEDS). TestOneHostReadPerStretch pins one host read per stretch. The sound beside a picture played apart is kept in the browser as far ahead as the picture has loaded and as far behind it as the picture holds, at most SOUND_MOST (180 s) each way (pictureSound.js: hls.js's goal and backBufferLength follow the picture's loaded and kept ranges), so a seek in the picture's loaded range finds its sound loaded too, as far as the browser holds of it: SOUND_BYTES (10 of the 12 MB it holds of a sound), which is seconds by the size of the sound's fragments as they come (the largest a second so far), two thirds of it ahead, the rest behind, SOUND_BACK at least. A 5.1 sound decoded as FLAC (some 500 KB a second) is some 14 s ahead and 10 behind; a stereo one, or one as it is, the picture's whole. That is the limit: a seek further than that, in the picture's loaded range, waits for the sound, which the core has cut and gives at once (an append past the browser's quota fails, and hls.js then lowers its goal for good, which the byte window avoids). A held picture looks again (toStart) whenever the browser says its ranges changed or it can play (progress, canplay, canplaythrough, loadeddata) and as hls.js says a fragment is buffered or appended: hls.js's word can come before the browser's ranges show it, and the picture would wait for ever on the last (Seeking into a part, the ranges late to show what hls.js buffered). Chromium raises progress rarely for a media source, so a backstop looks every two seconds (LOOKING, only while held) for the one case no event covers: the browser's ranges showing the last fragment late, with nothing more said (Seeking into a part, the ranges late and nothing said, the backstop). It goes to where it was sought only once what is loaded from the keyframe reaches there, not the start of the part it is in: hls.js's picture and sound each load their own part at the time the picture is at, their parts begin a few ms apart (the test film's sound's 28 ms before its picture's), and a part not yet loaded begins where hls.js guesses, so sought to a time between, the sound loaded the part after and never the one before, a hole behind the picture in what it waited to see loaded, and it waited for ever (#355's flake of the backstop story; Resuming in a part, between where its sound and its picture begin a part). Once what is loaded reaches the part it is in, it is not stepped on again where hls.js can load on to where it was sought from where the picture is: its goal is raised to that (lead, put back as it gets there), since each step is a seek, which decodes from the keyframe. Not past hls.js's cap, though (maxMaxBufferLength: AHEAD's 30 s, which hls.js halves after a full buffer): past it the goal does nothing, and the picture steps on, within what is loaded from the keyframe (Resuming in a part, hls.js held to load no further ahead). A held picture with nothing in flight and no step's seek under way counts as quiet (below: a step's seek is the decoder's, a second or more in a long stretch of a 4K film, and is not; Resuming in a part, its steps slow to decode), so one that waits on what never comes (a hole in what it has from the keyframe on, which hls.js, done loading ahead of it, does not go back for) is started again from the keyframe after five seconds, and said (hls.js went quiet: started again (the picture held in a part); Resuming in a part, a hole in what it waits for). A seek of the sound that found none of it loaded there, or took more than 300 ms to where the sound can play, is logged as a hitch, one in a few seconds (the sound beside the picture was sought, none of it loaded there: ready after N ms, at N s of the picture), with the picture's own a seek's own N ms stall beside it: the sound's wait is what the first says, and what is left of the second is the picture's decoder starting from its keyframe (a long GOP of a 4K film: nothing Mantle can change).

The server and its hosts, in full

Code: internal/server/ (routes.go, api.go, host.go, media.go), internal/art/.

server.New(core, data, host).Handler(ui) returns an http.Handler:

Path What it serves
/api/… The JSON API. One route for each call the frontend makes: the table is routes in internal/server/routes.go, each route answered by a small named handler (plugins.go, playback.go, library.go, profiles.go, backup.go, updates.go, together.go). A handler returns its answer and its error (handle, handleBody, handleRaw); a page that may wait for its providers is waits. Only a route that writes the response itself (a file, a picture, a range) takes the writer.
/art/{id}/{kind} Artwork (poster, backdrop) at the width asked for, resized once and cached in <data>/cache/art. A picture no page has shown for a month is let go of. A title id may be anything a plugin chose (the address has it escaped; its files are named by its hash if it is not plain letters, digits, -, _ and :). A picture over 40 MB is refused (422) and not fetched again; a title with no artwork is 404, a provider's failure 502.
everything else The frontend's files.

Host

What the surroundings do beyond the core is the Host interface. It has Kind (desktop, browser, page, or android for the Android app's window, which is a browser otherwise), OpenURL, Reveal, ChooseFolder, Window and Restart (the desktop's runs an update's program, or with none the app again; the server a browser opens ends, to be started again).

  • The desktop host opens links in the system's browser, shows files in the file browser, asks for a folder in the system's dialog, and drives the window: minimise, maximise, close, zoom.
  • server.Browser is the host in a browser. Its calls return ErrNotHere, and the API turns that into 501. Window is nil, so the window routes are 404.
  • server.Page is the web build's: a browser's, with nothing behind it to sign out of or download to.

GET /api/host tells the page which it is. The page then does what it can itself: a link opens in a tab, a download folder's path is typed, and the title bar's buttons and the zoom setting are hidden.

Media

server.Media(core, secret) serves the player what it cannot get elsewhere, under /media/<secret>/. The secret is new at every start, so nothing can read it by guessing.

  • /media/<secret>/<key>: the file of a finished download, with ranges for seeking.
  • /media/<secret>/relay/<key>: a stream that needs headers of its own (an addon's proxyHeaders). The player cannot set headers, so the core fetches the stream with them. It asks public addresses alone, on every hop of a redirect, unless that hop's host is one the stream's plugins may reach (Core.held, the one test every stream, download and frame read uses; a stream's own-host test comes with it from the resolver, resolved.own), and passes on only the headers a player reads (an addon cannot set a cookie on the app's address); every answer is nosniff and Content-Security-Policy: sandbox, never a page that runs there.

The desktop serves this on a listener of its own on 127.0.0.1 (ServeMedia). The Android app's server mounts it beside the API, behind the gate. The web build has none.

How the API answers errors

Code: internal/server/api.go (the statuses table).

replyOr answers a core error with the status that the statuses table in api.go gives it. Each row of the table is a status and the errors that map to it: unknown things are 404, conflicts are 409, bad input is 400 (including ErrNotShare and ErrNotPortable, a text that is not a setup or an export), a profile's PIN not given or wrong is 403 (ErrLocked, ErrWrongPIN), wrong too often is 429 (ErrWait, with Retry-After), as is a stream's host asking to be left alone (ErrBusy), the account's service refusing a mass deletion that looked like lost data is 428 (together.ErrMassDeletion, in together.Words' words), and what only the desktop can do (ErrNotHere) is 501. An error that is in no row is 500. To give a new core error a status, add it to its row. The body is the error's own text, and a toast shows it (api.said only takes off the request). A few pages map a status to words about their own call (the catalog's 409 and 400, the account's 404 for an ask's code, a restore's 428 (its round before refused a deletion, not the restore), a download's 409, the calendar's and a title's 404 as an empty state) and say so about that call alone; a core error is never replaced by another feature's words, so an error must say something true where it is returned. One made for a plugin (ErrNotListed, a repository's) is not returned from anywhere else. What is missing by an id that is not a title's is noSuch (ErrNoProfile, ErrNoRepository, ErrNoDevice, ErrNoFriend, ErrNoDownload, ErrNoFile, ErrNoBackend, ErrNoSource, ErrNoTrack, ErrNoEpisode, ErrNoShelf, ErrNoPlay), which says what it is and is ErrUnknown too, so it is answered as not found; a plugin is plugin.ErrUnknown. ErrUnknown itself is a title's, and the pictures a page fetches (which no toast shows). Other 404s say what they are in their own words, in the same row: ErrNotInList, ErrNotInContinue (a title with nothing to take out of Continue watching) and ErrOtherHistory (Undo of another profile's history). The title bar's buttons are a table too (windowActions).

Routes for sharing, export and backup

Code: internal/server/routes.go, backup.go, internal/core/export.go, frontend/src/bridge/api.js.

The routes for sharing and exporting:

Route What it does
GET /api/share The current profile's setup, to pick from (core.ShareView).
POST /api/share {pick} The code for what was picked, and the share it holds: {code, share}. With file: true it also writes the file to <data>/exports and adds its path.
GET /api/share/file?pick=… The same file, answered as a download. The pick, as JSON, is in the address.
POST /api/share/view {text} A pasted code or a file's text, as it would land in the current profile (ViewShare).
POST /api/share/apply {text, pick} Brings in what was picked of it, and answers what changed (ShareApplied).
POST /api/export The profile's library, lists and history, written to <data>/exports, answering its path.
GET /api/export/file The same file, answered as a download.
GET /api/copies The copies kept of the saved state (Copies).
POST /api/copies/restore {name} Replaces the state with one of them, then has the host restart Mantle ({restarting}).
POST /api/backup {passphrase, download} Everything as a backup file: written to <data>/exports, answering its path; with download: true held for one download, answering its url (GET /api/backup/file/{token}).
POST /api/backup/read {file} Whether a file is a backup this Mantle can open (ReadBackup).
POST /api/backup/open {file, passphrase} Opens it, holds it, answers {token, build, made, plugins}; a wrong passphrase is 403.
POST /api/app/restore/discard Takes away a restore that could not be finished as Mantle started.
POST /api/backup/restore {token} Replaces everything with what was opened, then has the host restart Mantle.

A file is saved one of two ways. The desktop's page POSTs, and the server writes the file with File.SaveIn into <data>/exports (API.save). A browser's page, the web build's and the Android app's WebView alike, fetches the GET route, which answers it as an attachment (download, with Content-Disposition), and hands the browser the answer as a blob under that name. saveFile in core/api.js chooses by the host's kind. In the Android app, the WebView's DownloadListener in MainActivity starts ACTION_CREATE_DOCUMENT, so the phone's own Save dialog asks where. The activity then fetches the address, signed in with the WebView's cookie, and writes the answer to the file the phone gave.

Restoring

Code: internal/core/restore.go, apply.go, copies.go, persist.go, cmd/mantle/update.go.

restore.go (the backup file and the stage), apply.go (applying it) and copies.go (the copies) put the saved state back from a copy persist.go keeps (Copies lists them: .bak, .before-*, .version-*, .unreadable-*, each with its time, its Build and its size, and Why it cannot be restored here, if it cannot: unreadable, or newer for a newer savedVersion; a file being written, ending in .tmp, is none; RestoreCopy) or from a backup file (BackupFile, ReadBackup, OpenBackup, RestoreBackup). Each copy is read as far as its first part says what it is (peek: saved puts version first and build second, so a state saved by this version has both there; one saved before, with build last, shows its version but no Mantle name, which the page says as "an older Mantle"). Both ways end in stage, then in ApplyRestore:

  • A backup is {backup, kdf, sealed}. sealed is a JSON content (build, made, state as mantle.json holds it (stateFile), account, keys) under AES-256-GCM, the data folder's own sealing (keyed, with sealBound: the header, Header.bound(), is authenticated beside it, so none of it can be changed or swapped). The key is the passphrase (normalised to NFC, at least 8 characters: BackupFile checks it) run through Argon2id (golang.org/x/crypto/argon2; 64 MiB, 4 passes, one lane, a salt per file: about 0.2 s natively, about 0.45 s in WebAssembly in Node; a phone's browser is a few times slower), the parameters being kdf in the file, so another version can use others and still open this one's. KDF.Valid bounds what a file may ask before anything is made: at most 128 MiB, 32 passes, and four times the app's work (memory times passes). A wrong passphrase, and a file that was changed, are the same ErrPassphrase. Nothing is read before it opens but the header (ReadBackup: ErrNotBackup, ErrNewer).
  • keys is every secret in the store but this device's own: deviceOwn in @together (its session, device, a pending asking, removed, and keys and holds, what the account lets it hold) is neither written to a backup nor replaced by a restore (withOwn); a device that has holds keeps all of its @together secrets, so it is not given the account's key. account (the account's id) is in the content, and a device signed in to an account takes only that account's backups (sameAccount, ErrOtherAccount: a backup made signed out is refused too); a device signed out takes any.
  • OpenBackup opens a file with its passphrase and says what it holds (BackupInfo: when, which Mantle, and the plugins installed from links that this device lacks, for the question); the API holds what it opened for two minutes under a token (API.hold), and RestoreBackup restores it once the person has confirmed, so the passphrase is made a key once and sent once.
  • readState is the one reading of a saved state (parseState is it for a state that is looked at, not loaded), for Load, the copies list and a restore: ErrNewer for a newer savedVersion, and also for one of this version that holds, in a strict field of what the user made (profiles, plugins, repositories, folders, downloads, titles), a value of a type this version cannot read (said as it is, to the person; status 426), ErrUnreadable for any other that is not this one's. What the file holds that this version has no place for comes with it (forward.go): replace writes the restored state with that put back (encodeKeeping), as any save does, and a backup is made from stateFile, the state as the file has it now, so a newer Mantle's additions go into it and come back out.
  • Staged. stage refuses what would remove a profile that has a PIN or change its PIN (pinsKept), keys that cannot be kept (Store.Kept) and another account's, then writes secrets.dat.restore (staged: an id of its own, the number of the restore (nextSeq), the state, the keys if any, the account; sealed with Store.Seal, as the keys are: the web build seals it as it seals secrets.dat, since seal.js covers secrets.dat.*), and the host restarts Mantle. A second restore replaces the stage. The running core carries on until it ends, and what it does and saves meanwhile is replaced.
  • Applied as Mantle starts (apply.go: ApplyRestore, called by start through openKeys, right after the keys are opened and before the state or the plugins are read, so nothing runs to race it). Applying is a function of the stage alone and runs again from its start whenever it did not finish, with the same result (applyStage): the state and keys it replaces are kept once for the restore, as mantle.json.before-restoring-<number> and secrets.dat.before-restoring-<number> (Store.KeepAs; not written again if there; the numbers count up, the last keptRestores are kept by number and not by time, so clocks that differ do not matter and the ones just made are never removed; keptCopies is the one scan for the next number and the pruning); the keys are replaced (Store.Replace, this device's own kept: read from the store as Mantle starts, so a session renewed after the confirmation is the one kept); mantle.json is written with the stage's id in it (saved.Restored), which says that this stage was made, then what sync keeps is made, by the same id (syncState.Restore, which says both that the plugins were queued and that the telling was made): the plugins to be installed again go to together.json's Waiting and synced profiles are told to give way to the account once (account.yieldAll), in one write, after the state, so a restore that did not finish has made neither. The stage is taken away after (removeStage). Nothing is undone on a failure, and, until the state is written, nothing else starts. The stage stays; ApplyRestore leaves the core stuck (Core.stuck, Recovery.Stuck) and start returns at once (started.idle: no state is read, no plugin loaded, nothing synced or listened for, no periodic save; stop and save do nothing), so that nothing moves under it (a session renewed, a plugin's token) before the person acts. The page asks GET /api/app/recovered first and shows RestoreFailed alone (main.js; the Close button on the desktop, which draws no title bar there). Try again is a restart (POST /api/app/restore/retry). Discard (DiscardRestore, which restarts Mantle after) puts the keys back from secrets.dat.before-restoring-<number> (Store.RestoreFrom: exact, as nothing ran since) and removes the stage. Once the state carries the stage's id the restore is made: a failure of what follows (what sync keeps, taking the stage away) is said once (Recovery.Restore, not stuck), Mantle starts, there is nothing to discard, and the steps run again at each start until they succeed (made by the stage's id, so none is done twice). A start gives way for a restore its sync state was not told of (the state has the id, and together.json neither Restore nor Gave of it: useRemote), and tells it by syncState.Gave, and by Restore too when no stage waits (secrets.dat.restore is gone: no plugins are left to queue, and builds before Gave read only Restore), so it gives way once, not at every start (what was changed offline since would be given up each time), while a stage that still waits queues its plugins (applyStage, which does not give way again for it). syncState.toldOf is the one reading: Restore or Gave. With a stage waiting only Gave is set, so the plugin step is not skipped, and a build before Gave finds the restore not told there: it gives way and queues the plugins as it did. A stage that cannot be opened says so as that (readStage).
  • Plugins installed from links are not in a backup as code, and nothing about them is the backup's to give. fateOf says, for one place of the plugins folder, which of the four a plugin of the restored state is: not from a link (as the backup has it), files here (this device's own entry stays, whatever the backup says: a restore of any kind never changes a plugin that is here), installed again (files not here: dropped from the state, recorded in Waiting with no grants, Shared.Name and Shared.Restored marking it) or not brought back (private to a profile, not here). OpenBackup lists the last two in the question (BackupInfo.Plugins, Private), from the same predicate. installWaiting (the household's retry with its backoff, which the sync loop now calls signed in or not) installs them through installSynced; for the entries marked Restored it gives up when the link answers without the plugin (ErrNotListed) and after restoredWaits, and syncState.forget keeps them when the household's are cleared (restoredOnly). The Plugins page shows them (Plugins.Restored) with Not here (DeclineHeld).
  • The restart is the host's (API.restarting: Host.Restart("")): the desktop's runs the app again (replace("") in cmd/mantle/update.go), the server a browser opens (restarter in cmd/mantle/web.go) ends and only the Android app's window starts it again (a server run by hand is not started again, and the page says so), and the web build has no process to restart, so its page reloads (restarted in app/pages/backups.js; the worker keeps what a request changed before it answers, so the stage is stored first).
  • A browser's backup is made by a POST that holds the file for one download at an address of its own (API.hold), so that the passphrase is in no address (the Android WebView's DownloadListener fetches an address); the desktop's is written to <data>/exports as an export is. A request about a backup may be as large as an import (maxImport).

Gate

Code: internal/server/gate.go, cmd/mantle/web.go.

The Android app's server wraps its handler in server.Gate. Without a session cookie, a request gets the sign-in page, or 401 for /api/, /art/ and /media/. A few files stay public, so the app can install: the manifest, the icons, the favicon and the service worker.

  • The Gate keeps its code, each signed-in device (the hash of its session token, and when it was last used) and an id in <data>/web-access.json. Up to 64 devices are kept (the oldest dropped), and one unused for 30 days is signed out on the server (its use is written down at most once a day); the cookie lasts as long. The id names the session cookie, mantle_<id>, because a browser's cookies belong to a host and not a port, so two servers on one computer each need their own. The cookie is HttpOnly and SameSite=Strict, and Secure over TLS.
  • CheckCode refuses a code given with -code that is not 8 to 64 letters or digits once dashes and spaces are taken out (ErrBadCode). cmd/mantle/web.go asks it before anything is opened, so a refused code does not make the data folder; OpenGate asks it too. A different code than the kept one drops every session.
  • Wrong codes are counted per address. From the fifth, each wait doubles from 2 seconds up to 10 minutes. An address is forgotten 15 minutes after its last wrong code, once its wait is over.
  • Gate.Handler wraps the door in http.NewCrossOriginProtection(), which refuses a request with an unsafe method (not GET, HEAD or OPTIONS) that comes from another site's page, signed in or not. It sets X-Frame-Options: DENY and Content-Security-Policy: frame-ancestors 'none' on every answer.
  • A request is answered only when its Host is an address (an IP, which a page elsewhere cannot rebind), localhost, this computer's own name (with .local), or a name given with -allow-host (Gate.Allow); any other name is 421, and the body names the Host it saw and the flag. That closes DNS rebinding, where another site's domain is pointed at this server and the cross-origin check sees the same name twice. /auth?code= signs in only a request from this computer itself (the Android window) by a loopback Host, so no code travels in a URL on a network, nor in one a reverse proxy here (all its clients are this computer) would log. The wrong-code slow-down is per address, an IPv6 address counted as its /64 network (one machine has all of it); with -proxied (Gate.Proxied) a request from this computer is counted by the last X-Forwarded-For it carries (Gate.client) and its cookie is Secure when the last X-Forwarded-Proto it carries is https (Gate.proxiedHTTPS), a header from a network is ignored, and a right code is never exempt (it would let guesses through at full speed). Signing out that cannot be written down answers 500, as the session would be back after a restart. -code is not the only way to give the access code: -code-file and MANTLE_CODE (server.GivenCode) keep it off the command line, which other programs on the computer can read; a code given by any of them is not printed at start. Nothing that changes state is a GET: finishing a pairing is POST /api/household/ask/answer.
  • CheckListen (called by cmd/mantle/web.go before the core or the data folder is opened) refuses an address that is not loopback unless -insecure-lan is given: this server has no TLS, so a network bind would send the code and the session in the clear.
  • cmd/mantle/web.go sets ReadHeaderTimeout to 10 seconds and IdleTimeout to 2 minutes, with no timeout for a whole request, since streams run long and pages wait on providers. It prints the access code with fmt.Fprintf(os.Stderr, …), not through log. start has sent log to the app's log file as well, and the Inspector shows that log, so the code is kept out of both. MainActivity leaves the line out of Android's log too.

Plugins in detail

Code: internal/plugin/, internal/script/, internal/core/scripts.go, settings.go.

Refresh and the Home layout

What a plugin offers can depend on its service. A plugin with a refresh function is asked to renew what it keeps in mantle.store when the app starts, when it is turned on, when it signs in or out (followSignIn, SignOut) and when its settings change (refreshPlugin in scripts.go), and is then read again, so its init makes its lists and shelves from that store. The core has nothing special for it: the shelves are ui.home.sections contributions like any other. The Jellyfin plugin keeps the user's libraries (/UserViews, in the order the user set, with LatestItemsExcludes) and offers a list and a shelf for each, library/<id>. The Home layout orders and hides them by those ids, and a library added to the server appears at the next refresh. When a plugin's refresh has succeeded, settleHome(id) (settings.go) puts the saved layout of every profile that has the plugin on and set up in terms of its shelves: a shelf the layout does not name goes where the shelves it replaces were (hidden if all were), else after the plugin's shelf before it, and ids with the plugin's prefix that none of its shelves has are dropped from homeOrder, homeHidden and homeShown. It is never run when the plugin is off, signed out or failed to refresh, nor for other plugins' ids, so what the layout holds then waits for the plugin. A refresh that could not list the libraries fails (after keeping what it can), so Mantle does not take them to be gone. A Jellyfin library shelf replaces the two mixed shelves the plugin had, and the server's address is kept with the libraries, so another server's are never shown.

The sandbox's network rules

Requests are made through the core's own network client. It does not follow redirects itself. A request a host sent on to another address is held to public addresses, so a redirect cannot reach the computer or its network. Artwork and subtitles a plugin passes on, relayed streams, previews, downloads and the frame reads of a stream are checked the same way on every hop (script.HoldUnless). A host is a plugin's when script.HostMatches says so, the one rule for scripts and the core: api.x.com, or *.x.com for its subdomains and x.com itself; a grant without a port is for any port, one with a port (host:8080) for that port only. A plugin's store counts its keys' names as well as its values against its 2 MB, and a key is at most 256 bytes (script.MaxKey). The addresses a user gives to install something (an addon's or a repository's link, which Core.download also refreshes from) may be on their network, but a hop after the first, to another host, is held (Core.held), so a public link cannot redirect into this computer's network.

A plugin that asks for the network and was refused it (Plugin.NetOff) is not Online. The core does not run its refresh (refreshPlugin), does not list it as an importer, and its health check (PluginHealth) answers one warning line that network access is off instead of the Lua error its script would end in. What the user asks of it besides (a sign-out) still runs, and a resolver's cancel is best effort.

Frontend notes

Code: frontend/src/app/ (router.js, speeds.js, stalledStream.js, video.js, party/), frontend/public/sw.js.

  • The router (router.js) is a stack of routes in memory. A page's scroll and focus are kept by its history entry (a route is its own entry; a main section chosen again inherits what it kept), so Back returns you to exactly where you were, and a page opened again as a new visit starts at its top (held by its anchor shelf until nothing in it is still loading), and a page opened anew starts at its top (toTop, before the page's own effects, so one opened to a place keeps to it). A returning page whose data is remembered is in place before it is first drawn: restore runs as a layout effect, so it is never drawn at its top for a frame and then moved (with animations off that frame is seen). One with nothing remembered (a title not seen lately, or after a profile switch) first shows its skeleton, its top element aria-busy; restore waits for that by animation frame and puts the page back once it is not busy, where hold keeps it. In a browser it follows history (followHistory), so the browser's Back is the app's. Each entry says which run of the app's history it is of; a reload or a profile switch starts a new run, and a Back that lands on an entry of another run passes over it. A menu or sheet that opens takes an entry of its own (holdBack, called from App.js), so Back closes it first, even on the first page, where Back would otherwise leave the app.
  • The speed is the viewer's, or in a party the host's: nothing measures the device or holds a picture below what is asked, and a device or a network that cannot keep up buffers or drops frames, as any player does. (pace.js used to measure the device's throughput in pixels a second and hold the speed to a step under it, with a notice, a capped speed menu and tries every two minutes; issue #288 removed it, as a limit measured on one film capped another, even of the same codec.) app/speeds.js holds the offered speeds (quarter steps up to 5×), off (whether a picture is apart from its sound by 0.35 s of the playing) and REALIGN.
  • The log keeps the picture's hitches and its summary (video/watch.js): a stall of a second or more (with where, the speed and size, how much was loaded ahead, and at the playhead where the seek bar's reckoning says more, the frames dropped since and the network's state), a seek's own stall, a picture stuck 700 ms or more, a page held up, and, for a minute or more of play, how long it played, at what speeds, its stalls, holds for its sound, realignments and frames dropped. Frames are read with requestVideoFrameCallback; without it only the stalls the WebView says are logged.
  • A picture is sought to its sound (realign in video/watch.js) not more than once in REALIGN (20 s), when it is apart from its sound for a second (off), after a stall that sped up, or when it is stuck while the sound plays on; never one played apart (pictureSound.js), and not on standby, paused or seeking: a seek decodes from a keyframe, so it cannot make a slow decoder faster. A stall that was a seek's own (the seek still on, nothing shown since it, or the start) is logged as that (a seek's own N ms stall), is not a hitch, and asks for no seek.
  • In a watch party the host reports the speed it plays at (rate; asked is sent with the same value for older builds), the page's body read whole by core.PartyPlay, and every player plays at it, whatever its device: a guest follows it (follow in party/). A guest off the host's place by more than DRIFT is moved to it, not more than once in REALIGN seconds (at once where the host sought or skipped, or where the guest came back from away, and at joining); a little off, it plays a little faster or slower until it is back. Catch up is the notice of a guest that stepped away. A guest that buffers holds the party (the host's player pauses for it), so it does not fall behind for that.
  • A repackaged stream is watched for hls.js gone quiet (stalledStream.js, video.js): hls.js 1.7 asks for the init segment again with each try of a part and aborts the one on its way, so a part the core refuses while the init segment is slow to come leaves it loading nothing, with the picture not playing, and nothing else guards that: hls.js's media source has the media's length from the playlist with nothing loaded (the picture reads duration above 0 at readyState 0), so the player counts the stream as started and its 45 s give-up for a stream that does not start has nothing to do (RepackagedQuiet holds that timer and asserts it fails nothing). The report to make upstream (issue #87) is in PR #99's description. Every second the picture is sampled, and a second is quiet when nothing is in flight (the count of the loader hls.js loads through, counted), the picture has not ended, and it is held to step through a part with no step's seek under way (it waits on hls.js, whatever the browser could play) or cannot play (readyState under 3) with under half a second of the film at the playhead (reach: a seek or a stall on a device that decodes slowly has its buffer full and is a decoder at work, and a part that does not start with a keyframe shows nothing loaded on its own). Five quiet seconds in a row make hls.js anew, from where the picture is wanted (wanted: where it was to start, or a seek made meanwhile, which a destroyed picture, at 0, does not say); five more before it plays again (playing puts the watch back to its first step, as it does the stuck one's: quietAgain, Repackaged, gone quiet, started again and played on) fail the source if it has not played (played, which a late switch to hls.js does not clear, and the player's own record of it, one for each stream for the playback, written only as that stream's picture is heard (a reconnect's new picture, or the source picked again after another, does not lose it, and a standby gets it too): the player is told, onError (a request that fails is asked again on a later tick), and the core fails over as for a stream that does not start; while another picture is to take over the player drops that, so the picture is left as it is, not let go, and asked again every five quiet seconds until the player acts, which makes it another picture) and else play the stream as it is, which keeps the picture and loses the audio tracks and a sound only the core decodes. A picture that cannot play with its film at the playhead is not quiet but can be stuck (stuckSeconds, issue #287: a decoder or hls.js's media source that stopped with its film in hand, 42 s with 25 s loaded, once went unrecovered because the watch only looked at what was not coming): a second counts when the playhead has not moved since the last, the picture cannot play, has not ended, is not held and has half a second or more of film at the playhead, whatever is in flight (a decoder that is only slow moves the playhead, and is not counted). Five such seconds nudge it (currentTime set to where it is: the decoder starts again from its keyframe, nothing read of the host), five more start the stream again where it is (hls.js anew, the plain stream loaded again, the picture played apart made again: restart), five more act as a quiet one does (fail the source if nothing has played, else play the stream as it is, a plain one failing the source and being asked again every five seconds); playing again puts it back to the first step. The log's stalls say how much was loaded ahead as the WebView's range at the playhead has it too (n s at the playhead) where the seek bar's reckoning from bytes says more. That plain stream has a watch of its own, by networkState (loading is in flight), which fails the source the same way and asks again every five quiet seconds until the player acts, as the repackaged one does; it stops once hls.js is in place. Each step is said to the log once, a picture on standby's by the player (the picture to take over: …). quiet.test.js tests the decision and the loader's count in Node; the stories of PlayerStreams hold the watch's seconds with heldTimers, and some play on the stand-ins of test/media.js.
  • app/stalledStream.test.js tests the quiet and stuck counts and the loader's count of loads, and which pictures are watched (playing), in Node. The stories of PlayerStreams* hold the watch's seconds with heldTimers and play on the stand-ins of testkit/media.js (a stuck film, a hanging seek); PlayerParty holds that a guest plays at the host's speed.
  • sw.js keeps the page and its files so a browser opens the app without a connection. It never keeps the API, artwork, media or the sign-in page. Registered with ?page (the web build), it passes /api/ and /art/ to the page (see The web build), and keeps what the page says it loaded before the worker had it. Its paths are relative to where it is, so the site works at any address.

Folders and smaller notes

Folder What it holds
src/main.js Starts the app: input tracking, the window, /api/host, then renders. In a browser it also follows the browser's history and registers the service worker (not in the Android app's window, kind android, whose address is new at each launch).
src/bridge/ api.js (every call into the core, over fetch), host.js (desktop or browser), window.js (the native window), copy.js (copy text to the clipboard).
src/lib/ html.js (htm bound to Preact, with small helpers) and words.js (how sentences name things: listed, lower, count).
src/app/ The shell (App.js), the rail (rail.js) and the phone's tab bar (tabs.js), the router, the player, navigation, settings and downloads state.
src/app/pages/ One file for each screen.
src/ui/ The component kit. Each has Storybook stories.
src/nav/ Focus movement for keyboard and controller.
src/input/ Input modality (keys, mouse, touch), the controller, long press and hold.
src/styles/ Design tokens, themes and base styles.
src/testkit/ A stand-in core for stories, and the stories' helpers (clock.js, sim.js).
src/fixtures/ What the stories and tests read: JSON the core answers with, clips, HLS files.
src/web/ The web build's page side: core.js (starting the core, one tab at a time), worker.js (the core's worker), files.js (its files).
public/ The manifest, the icons and sw.js, for a browser.
  • Almost every call into the core goes through core/api.js. In the Android app's WebView, a 401 reloads the page, which signs in again.
  • core/window.js calls /api/window/… for the title bar's buttons. The only direct native calls are window.chrome.webview.postMessage('wails:drag') and 'wails:resize:…', which the desktop window understands.
  • core/copy.js copies with the Clipboard API where a page may use it (the desktop app, https, this computer's own address). Elsewhere, such as over a network's plain http, it copies a selection of a hidden text box and puts focus back. It answers whether the copy worked.
  • lib/words.js is for sentences the app says: listed(['A', 'B', 'C']) is "A, B and C", lower('Server') is "server" (and leaves "API key" alone), count(2, 'title') is "2 titles".
  • useLoad loads a page's data, asks again while parts are pending, remembers what a page last showed, and asks again quietly when the core says something is new.
  • The phone layout is CSS and ui/phone.js. At 760 pixels wide or less, or on a touch screen 500 pixels tall or less (a phone held sideways), the rail (rail.js) becomes a tab bar (tabs.js), menus become sheets and the player is driven by touch. The media query is written out in phone.js and in each stylesheet that has a phone layout. The tab bar and the top bar use env(safe-area-inset-*) to keep clear of a notch.
  • The desktop's custom title bar is the page's top bar, marked as the window's drag area.

Plugin state

State that belongs to the device is shared: what a plugin was granted, its settings and its place in the priority order. Whether it is on is each profile's (Plugin.Chosen, by profile id), else how it starts (Plugin.On): one that comes with the app, as it comes or as the first run set it; one installed from a link, off but for the profile that installed it (Host.Add's by; none in the first run, so on for all). The host's List and Get give a profile's view, its own choice as on. A plugin made private is visible to one profile only.

This page is made from docs/dev/internals.md in the repository.