Encoding

Loro 1.0 has stabilized the data format and will not have any breaking changes.

Loro introduces three encoding modes to meet the needs for different use cases:

Updates Encoding: Encodes all or a specified range of operations, mainly used for delta updates of documents.
Snapshot Encoding: Encodes the entire document, including both all the operations and the current document state.
Shallow Snapshot Encoding: Loro 1.0 provides a new encoding format for discarding useless historical operations. It is a special snapshot that encodes the most recent historical operations, the starting state of the remaining history, and the latest state of the document.

Different encoding formats are provided through the unified doc.export(mode), and all binary encoding formats can be imported via doc.import(bytes).

export type ExportMode =
  | {
      mode: "update";
      from?: VersionVector;
    }
  | {
      mode: "updates-in-range";
      spans: {
        id: ID;
        len: number;
      }[];
    }
  | {
      mode: "snapshot";
    }
  | {
      mode: "shallow-snapshot";
      frontiers: Frontiers;
    };

Updates Encoding

There are two modes for updates encoding. update 和 updates-in-range.

update mode will encode all ops from the from version to the latest version of the document.
updates-in-range mode will encode all ops within the specified version range.

const doc1 = new LoroDoc();
const doc2 = new LoroDoc();
doc2.setPeerId(2);
 
doc2.getText("text").insert(0, "hello");
const bytes = doc2.export({ mode: "update", from: doc1.version() }); // Uint8Array
// const bytes = doc2.export({ mode: "updates-in-range", spans: [{id: { peer: 2, counter: 0 }, len: 1}] });
// These bytes can be stored in a database or transmitted over a network.
doc1.import(bytes);
console.log(doc1.toJSON()); // { text: "hello" }

Updates Encoding selectively encodes operations from a chosen initial version to the most recent, enhancing support for real-time collaboration by focusing on incremental changes rather than the entire document state.

Snapshot Encoding

const doc1 = new LoroDoc();
const doc2 = new LoroDoc();
 
doc2.getText("text").insert(0, "hello");
const bytes = doc2.export({ mode: "snapshot" }); // Uint8Array
// These bytes can be stored in a database or transmitted over a network.
doc1.import(bytes);
console.log(doc1.toJSON()); // { text: "hello" }

Snapshot Encoding comprehensively captures a document's current state and its historical operations. This mode can quickly obtain the latest document state.

Shallow Snapshot Encoding

Loro will save all editing history to resolve conflicts and history backtracking. However, for most scenes, most of the older history is useless, taking up too much extra memory and requires more storage space for saving.

Loro 1.0 provides a shallow snapshot encoding mode. You can specify the starting historical version to be retained, and Loro will truncate all the history before this version.

const doc1 = new LoroDoc();
const doc2 = new LoroDoc();
doc2.setPeerId(2);
 
const text = doc2.getText("text");
text.insert(0, "hello");
const frontier = doc2.frontiers();
text.insert(0, "w");
text.insert(0, "o");
text.insert(0, "r");
text.insert(0, "l");
text.insert(0, "d");
const bytes = doc2.export({ mode: "shallow-snapshot", frontiers }); // Uint8Array
// These bytes can be stored in a database or transmitted over a network.
doc1.import(bytes);
console.log(doc1.toJSON()); // { text: "hello" }

Note: CRDTs require information about the history of Ops to resolve conflicts, and the limitation of using shallow snapshots is that new updates that depend on truncated historical ops cannot be imported any more.

Best Practices for Persisting Loro Documents

Use Updates Encoding to export delta updates with each keystroke, facilitating immediate synchronization.
If you need to quickly load the latest state of the document to provide to the user, then you need Snapshot Encoding.
If you think your LoroDoc has grown large and older history is safe to recycle, you can use Shallow Snapshot Encoding.

For collaborative purposes, the binary data generated by Snapshot/Updates Encoding can be transmitted through any medium, such as WebRTC, WebSocket, or HTTP.

The strong eventual consistency in CRDTs ensures that peers with identical sets of operations will converge to the same document state, obviating concerns about the order, duplication, or timing of operation delivery.

Composing CRDTs Cursor