JS/WASM Benchmarks
The benchmark is based on crdt-benchmarks (opens in a new tab) repo.
- The
memUsedresult has been removed from the current benchmark because it is not reported correctly when using WASM. - During the benchmark,
loro-wasmwas configured not to use compression on the doc, and it preserves the entire editing history, resulting in a largerdocSize.
On 2023-11-08, the benchmark test was performed on a 13-inch M1 MacBook Pro 2020.
The benchmark uses the most recent versions of loro-wasm, ywasm, and yjs available
on that date. However, an older version of automerge-wasm (v0.1.2, published in 2022)
was used.
NOTE: The current benchmark is somewhat outdated, particularly after numerous refactorings, which have altered performance characteristics. We plan to update this document shortly.
Benchmark setup
B1: No conflicts
Simulate two clients. One client modifies a text object and sends update
messages to the other client. We measure the time to perform the task (time),
the amount of data exchanged (avgUpdateSize), the size of the encoded document
after the task is performed (docSize), the time to parse the encoded document
(parseTime), and the memory used to hold the decoded document (memUsed).
B2: Two users producing conflicts
Simulate two clients. Both start with a synced text object containing 100
characters. Both clients modify the text object in a single transaction and then
send their changes to the other client. We measure the time to sync concurrent
changes into a single client (time), the size of the update messages
(updateSize), the size of the encoded document after the task is performed
(docSize), the time to parse the encoded document (parseTime), and the
memory used to hold the decoded document (memUsed).
B3: Many conflicts
Simulate √N concurrent actions. We measure the time to perform the task and
sync all clients (time), the size of the update messages (updateSize), the
size of the encoded document after the task is performed (docSize), the time
to parse the encoded document (parseTime), and the memory used to hold the
decoded document (memUsed). The logarithm of N was chosen because √N
concurrent actions may result in up to √N^2 - 1 conflicts (apply action 1: 0
conlict; apply action2: 1 conflict, apply action 2: 2 conflicts, ..).
B4: Real-world editing dataset
Replay a real-world editing dataset. This dataset contains the character-by-character editing trace of a large-ish text document, the LaTeX source of this paper: https://arxiv.org/abs/1608.03960 (opens in a new tab)
Source: https://github.com/automerge/automerge-perf/tree/master/edit-by-index (opens in a new tab)
- 182,315 single-character insertion operations
- 77,463 single-character deletion operations
- 259,778 operations totally
- 104,852 characters in the final document
We simulate one client replaying all changes and storing each update. We measure
the time to replay the changes and the size of all update messages
(updateSize), the size of the encoded document after the task is performed
(docSize), the time to encode the document (encodeTime), the time to parse
the encoded document (parseTime), and the memory used to hold the decoded
document in memory (memUsed).
[B4 x 100] Real-world editing dataset 100 times
Replay the [B4] dataset one hundred times. The final document has a size of over 10 million characters. As comparison, the book "Game of Thrones: A Song of Ice and Fire" is only 1.6 million characters long (including whitespace).
- 18,231,500 single-character insertion operations
- 7,746,300 single-character deletion operations
- 25,977,800 operations totally
- 10,485,200 characters in the final document
| N = 6000 | loro-wasm | yjs | automerge-wasm | ywasm |
|---|---|---|---|---|
| [B1.1] Append N characters (time) | 54 ms | 86 ms | 61 ms | 72 ms |
| [B1.1] Append N characters (avgUpdateSize) | 58 bytes | 27 bytes | 121 bytes | 27 bytes |
| [B1.1] Append N characters (encodeTime) | 0 ms | 1 ms | 7 ms | 0 ms |
| [B1.1] Append N characters (docSize) | 6219 bytes | 6031 bytes | 3995 bytes | 6031 bytes |
| [B1.1] Append N characters (parseTime) | 0 ms | 0 ms | 38 ms | 0 ms |
| [B1.2] Insert string of length N (time) | 0 ms | 1 ms | 28 ms | 0 ms |
| [B1.2] Insert string of length N (avgUpdateSize) | 6096 bytes | 6031 bytes | 6201 bytes | 6031 bytes |
| [B1.2] Insert string of length N (encodeTime) | 0 ms | 2 ms | 2 ms | 0 ms |
| [B1.2] Insert string of length N (docSize) | 6148 bytes | 6031 bytes | 3977 bytes | 6031 bytes |
| [B1.2] Insert string of length N (parseTime) | 0 ms | 0 ms | 12 ms | 0 ms |
| [B1.3] Prepend N characters (time) | 23 ms | 73 ms | 45 ms | 16 ms |
| [B1.3] Prepend N characters (avgUpdateSize) | 57 bytes | 27 bytes | 116 bytes | 27 bytes |
| [B1.3] Prepend N characters (encodeTime) | 1 ms | 2 ms | 8 ms | 0 ms |
| [B1.3] Prepend N characters (docSize) | 6165 bytes | 6041 bytes | 3991 bytes | 6041 bytes |
| [B1.3] Prepend N characters (parseTime) | 1 ms | 5 ms | 34 ms | 1 ms |
| [B1.4] Insert N characters at random positions (time) | 19 ms | 68 ms | 214 ms | 80 ms |
| [B1.4] Insert N characters at random positions (avgUpdateSize) | 58 bytes | 29 bytes | 121 bytes | 29 bytes |
| [B1.4] Insert N characters at random positions (encodeTime) | 1 ms | 2 ms | 8 ms | 0 ms |
| [B1.4] Insert N characters at random positions (docSize) | 29503 bytes | 29554 bytes | 24746 bytes | 29554 bytes |
| [B1.4] Insert N characters at random positions (parseTime) | 1 ms | 3 ms | 64 ms | 3 ms |
| [B1.5] Insert N words at random positions (time) | 20 ms | 71 ms | 986 ms | 245 ms |
| [B1.5] Insert N words at random positions (avgUpdateSize) | 63 bytes | 36 bytes | 131 bytes | 36 bytes |
| [B1.5] Insert N words at random positions (encodeTime) | 2 ms | 3 ms | 23 ms | 1 ms |
| [B1.5] Insert N words at random positions (docSize) | 98899 bytes | 87924 bytes | 96206 bytes | 87924 bytes |
| [B1.5] Insert N words at random positions (parseTime) | 2 ms | 8 ms | 144 ms | 5 ms |
| [B1.6] Insert string, then delete it (time) | 0 ms | 5 ms | 36 ms | 0 ms |
| [B1.6] Insert string, then delete it (avgUpdateSize) | 6191 bytes | 6053 bytes | 6338 bytes | 6053 bytes |
| [B1.6] Insert string, then delete it (encodeTime) | 0 ms | 0 ms | 4 ms | 0 ms |
| [B1.6] Insert string, then delete it (docSize) | 6143 bytes | 38 bytes | 3996 bytes | 38 bytes |
| [B1.6] Insert string, then delete it (parseTime) | 0 ms | 0 ms | 28 ms | 0 ms |
| [B1.7] Insert/Delete strings at random positions (time) | 28 ms | 85 ms | 763 ms | 74 ms |
| [B1.7] Insert/Delete strings at random positions (avgUpdateSize) | 61 bytes | 31 bytes | 135 bytes | 31 bytes |
| [B1.7] Insert/Delete strings at random positions (encodeTime) | 1 ms | 7 ms | 18 ms | 0 ms |
| [B1.7] Insert/Delete strings at random positions (docSize) | 51470 bytes | 28377 bytes | 59284 bytes | 28377 bytes |
| [B1.7] Insert/Delete strings at random positions (parseTime) | 1 ms | 10 ms | 112 ms | 2 ms |
| [B1.8] Append N numbers (time) | 23 ms | 85 ms | 146 ms | 16 ms |
| [B1.8] Append N numbers (avgUpdateSize) | 60 bytes | 32 bytes | 125 bytes | 32 bytes |
| [B1.8] Append N numbers (encodeTime) | 2 ms | 1 ms | 10 ms | 0 ms |
| [B1.8] Append N numbers (docSize) | 47623 bytes | 35634 bytes | 26988 bytes | 35634 bytes |
| [B1.8] Append N numbers (parseTime) | 3 ms | 0 ms | 68 ms | 0 ms |
| [B1.9] Insert Array of N numbers (time) | 5 ms | 1 ms | 24 ms | 2 ms |
| [B1.9] Insert Array of N numbers (avgUpdateSize) | 35725 bytes | 35657 bytes | 31199 bytes | 35657 bytes |
| [B1.9] Insert Array of N numbers (encodeTime) | 1 ms | 1 ms | 2 ms | 0 ms |
| [B1.9] Insert Array of N numbers (docSize) | 47648 bytes | 35657 bytes | 26956 bytes | 35657 bytes |
| [B1.9] Insert Array of N numbers (parseTime) | 3 ms | 0 ms | 23 ms | 0 ms |
| [B1.10] Prepend N numbers (time) | 16 ms | 51 ms | 382 ms | 15 ms |
| [B1.10] Prepend N numbers (avgUpdateSize) | 59 bytes | 32 bytes | 120 bytes | 32 bytes |
| [B1.10] Prepend N numbers (encodeTime) | 1 ms | 1 ms | 10 ms | 0 ms |
| [B1.10] Prepend N numbers (docSize) | 47643 bytes | 35665 bytes | 26990 bytes | 35665 bytes |
| [B1.10] Prepend N numbers (parseTime) | 3 ms | 1 ms | 51 ms | 2 ms |
| [B1.11] Insert N numbers at random positions (time) | 16 ms | 58 ms | 458 ms | 113 ms |
| [B1.11] Insert N numbers at random positions (avgUpdateSize) | 62 bytes | 34 bytes | 125 bytes | 34 bytes |
| [B1.11] Insert N numbers at random positions (encodeTime) | 1 ms | 2 ms | 12 ms | 0 ms |
| [B1.11] Insert N numbers at random positions (docSize) | 70903 bytes | 59137 bytes | 47749 bytes | 59137 bytes |
| [B1.11] Insert N numbers at random positions (parseTime) | 3 ms | 3 ms | 70 ms | 3 ms |
| [B2.1] Concurrently insert string of length N at index 0 (time) | 0 ms | 0 ms | 104 ms | 0 ms |
| [B2.1] Concurrently insert string of length N at index 0 (updateSize) | 9256 bytes | 6094 bytes | 9499 bytes | 6093 bytes |
| [B2.1] Concurrently insert string of length N at index 0 (encodeTime) | 0 ms | 0 ms | 4 ms | 0 ms |
| [B2.1] Concurrently insert string of length N at index 0 (docSize) | 12282 bytes | 12152 bytes | 8014 bytes | 12150 bytes |
| [B2.1] Concurrently insert string of length N at index 0 (parseTime) | 0 ms | 0 ms | 43 ms | 0 ms |
| [B2.2] Concurrently insert N characters at random positions (time) | 117 ms | 25 ms | 1496 ms | 210 ms |
| [B2.2] Concurrently insert N characters at random positions (updateSize) | 344345 bytes | 33444 bytes | 1093293 bytes | 177007 bytes |
| [B2.2] Concurrently insert N characters at random positions (encodeTime) | 1 ms | 2 ms | 18 ms | 1 ms |
| [B2.2] Concurrently insert N characters at random positions (docSize) | 59356 bytes | 66860 bytes | 50708 bytes | 66852 bytes |
| [B2.2] Concurrently insert N characters at random positions (parseTime) | 2 ms | 15 ms | 142 ms | 6 ms |
| [B2.3] Concurrently insert N words at random positions (time) | 144 ms | 55 ms | 2915 ms | 569 ms |
| [B2.3] Concurrently insert N words at random positions (updateSize) | 408728 bytes | 88994 bytes | 1185202 bytes | 215213 bytes |
| [B2.3] Concurrently insert N words at random positions (encodeTime) | 3 ms | 4 ms | 48 ms | 4 ms |
| [B2.3] Concurrently insert N words at random positions (docSize) | 197284 bytes | 178130 bytes | 191500 bytes | 178130 bytes |
| [B2.3] Concurrently insert N words at random positions (parseTime) | 3 ms | 13 ms | 311 ms | 11 ms |
| [B2.4] Concurrently insert & delete (time) | 259 ms | 95 ms | 5682 ms | 1500 ms |
| [B2.4] Concurrently insert & delete (updateSize) | 786130 bytes | 139517 bytes | 2395876 bytes | 398881 bytes |
| [B2.4] Concurrently insert & delete (encodeTime) | 4 ms | 6 ms | 87 ms | 5 ms |
| [B2.4] Concurrently insert & delete (docSize) | 304590 bytes | 279166 bytes | 307364 bytes | 279172 bytes |
| [B2.4] Concurrently insert & delete (parseTime) | 4 ms | 27 ms | 444 ms | 32 ms |
| [B3.1] 20√N clients concurrently set number in Map (time) | 23 ms | 52 ms | 35 ms | 156 ms |
| [B3.1] 20√N clients concurrently set number in Map (updateSize) | 63850 bytes | 49163 bytes | 283296 bytes | 49172 bytes |
| [B3.1] 20√N clients concurrently set number in Map (encodeTime) | 2 ms | 2 ms | 13 ms | 1 ms |
| [B3.1] 20√N clients concurrently set number in Map (docSize) | 38464 bytes | 32212 bytes | 83205 bytes | 32214 bytes |
| [B3.1] 20√N clients concurrently set number in Map (parseTime) | 22 ms | 14 ms | 38 ms | 10 ms |
| [B3.2] 20√N clients concurrently set Object in Map (time) | 40 ms | 50 ms | 42 ms | 156 ms |
| [B3.2] 20√N clients concurrently set Object in Map (updateSize) | 99763 bytes | 85077 bytes | 325370 bytes | 85083 bytes |
| [B3.2] 20√N clients concurrently set Object in Map (encodeTime) | 3 ms | 2 ms | 20 ms | 1 ms |
| [B3.2] 20√N clients concurrently set Object in Map (docSize) | 74377 bytes | 32223 bytes | 90426 bytes | 32240 bytes |
| [B3.2] 20√N clients concurrently set Object in Map (parseTime) | 24 ms | 13 ms | 34 ms | 11 ms |
| [B3.3] 20√N clients concurrently set String in Map (time) | 100 ms | 70 ms | 215 ms | 167 ms |
| [B3.3] 20√N clients concurrently set String in Map (updateSize) | 7840873 bytes | 7826231 bytes | 8063440 bytes | 7826231 bytes |
| [B3.3] 20√N clients concurrently set String in Map (encodeTime) | 28 ms | 2 ms | 52 ms | 2 ms |
| [B3.3] 20√N clients concurrently set String in Map (docSize) | 7815449 bytes | 38369 bytes | 95086 bytes | 36840 bytes |
| [B3.3] 20√N clients concurrently set String in Map (parseTime) | 7 ms | 13 ms | 74 ms | 14 ms |
| [B3.4] 20√N clients concurrently insert text in Array (time) | 28 ms | 44 ms | 33 ms | 165 ms |
| [B3.4] 20√N clients concurrently insert text in Array (updateSize) | 70476 bytes | 52745 bytes | 285330 bytes | 52737 bytes |
| [B3.4] 20√N clients concurrently insert text in Array (encodeTime) | 1 ms | 2 ms | 19 ms | 1 ms |
| [B3.4] 20√N clients concurrently insert text in Array (docSize) | 47868 bytes | 26590 bytes | 83577 bytes | 26582 bytes |
| [B3.4] 20√N clients concurrently insert text in Array (parseTime) | 5 ms | 2 ms | 41 ms | 4 ms |
| [B4] Apply real-world editing dataset (time) | 158 ms | 1028 ms | 1542 ms | 23006 ms |
| [B4] Apply real-world editing dataset (avgUpdateSize) | skipped | 29 bytes | skipped | skipped |
| [B4] Apply real-world editing dataset (encodeTime) | 2 ms | 4 ms | 131 ms | 3 ms |
| [B4] Apply real-world editing dataset (docSize) | 260815 bytes | 159929 bytes | 129098 bytes | 159929 bytes |
| [B4] Apply real-world editing dataset (parseTime) | 2 ms | 6 ms | 615 ms | 14 ms |
| [B4x100] Apply real-world editing dataset 100 times (time) | 15251 ms | 110404 ms | skipped | skipped |
| [B4x100] Apply real-world editing dataset 100 times (encodeTime) | 222 ms | 266 ms | skipped | skipped |
| [B4x100] Apply real-world editing dataset 100 times (docSize) | 26826425 bytes | 15989245 bytes | skipped | skipped |
| [B4x100] Apply real-world editing dataset 100 times (parseTime) | 180 ms | 1357 ms | skipped | skipped |