Chia Proof of Space Construction

Introduction

In order to create a secure blockchain consensus algorithm using disk space, a Proof of Space is scheme is necessary. This document describes a practical contruction of Proofs of Space, based on Beyond Hellman’s Time-Memory Trade-Offs with Applications to Proofs of Space [1]. We use the techniques laid out in that paper, extend it from 2 to 7 tables, and tweak it to make it efficient and secure, for use in the Chia Blockchain. The document is divided into three main sections: What (mathematical definition of a proof of space), How (how to implement proof of space), and Why (motivation and explanation of the construction) sections. The Beyond Hellman paper can be read first for more mathematical background.

• Chia Proof of Space Construction
  • Introduction
  • What is Proof of Space?
    • Definitions
    • Proof format
    • Proof Quality String
    • Definition of parameters, and $\mathbf{M}, f, \mathcal{A}, \mathcal{C}$ functions:
      • Parameters:
      • $f$ functions:
      • Matching function $\mathbf{M}$:
      • $\large \mathcal{A'}$ function:
      • $\large \mathcal{A_t}$ function:
      • Collation function $\mathcal{C}$:
  • How do we implement Proof of Space?
    • Plotting
      • Plotting Tables (Concepts)
        • Tables
        • Table Positions
        • Compressing Entry Data
        • Delta Format
        • ANS Encoding of Delta Formatted Points
        • Stub and Small Deltas
        • Parks
        • Checkpoint Tables
      • Plotting Algorithm
        • Final Disk Format
        • Full algorithm
        • Phase 1: Forward Propagation
        • Phase 2: Backpropagation
        • Phase 3: Compression
        • Phase 4: Checkpoints
        • Sort on Disk
        • Plotting Performance
        • Space Required
    • Proving
      • Proof ordering vs Plot ordering
      • Proof Retrieval
      • Quality String Retrieval
      • Proving Performance
    • Verification
  • Construction Explanation (Why)
    • Blockchain consensus summary
    • AACKPR17
    • 7 Tables
    • Collation of Inputs
    • Inlining Islands
    • Matching Function Requirements
    • Minimum and maximum plot sizes
    • Expected number of proofs
    • Using AES256 and RRAES128
    • Fx custom mode
    • Quality String
    • Plot seed
    • Plotting algorithm cache locality
  • Potential Optimizations
    • Variable Sized Parks
    • Cycles attack
    • Hellman Attacks
    • Dropping bits in x values / line points
    • Reduce Plotting Space overhead
    • Mixing passes
    • Every pos_R value is used at least once
    • Pruning bad x values
    • Replotting Attack
    • Reduce Checkpoint table size
    • Faster Disk
    • Multithreading
  • References

What is Proof of Space?

Definitions

• $[X]$ denotes the set $\{0, 1, ..., X-1\}$
• $AES_{256}(c, K) : [2^{128}] \rightarrow [2^{128}]$ is a 14 (13+1) round AES256 encryption with a 256-bit key $K$.
• $AES_{128}(c; K) : [2^{128}] \rightarrow [2^{128}]$ is a 2 (2+0) round AES128 encryption with a 128-bit key $K$
• $e$ is the natural logarithm
• $\ll, \gg$ are bitwise left-shift and right-shift operators
• $\%$ is the modulus operator
• $\text{divmod}(x, m) = (\lfloor \frac{x}{m} \rfloor, x \% m)$
• $\oplus$ denotes bitwise-XOR operation
• $x \mathbin{\|} y$ denotes zero-padded concatenation: for the implied domain $y \in [2^z]$, it is $(x \ll z) + y$
• $\underset{b}{\text{trunc}}(x)$ = the first (most significant) $b$ bits of $x$. If $x \in [2^z]$ is the implied domain, equal to $x \gg (z - b)$.
• $x[a:b]$ If $x$ has implied domain $[2^z]$, then this is the $a^{\text{th}}$ to $(b-1)^{\text{th}}$ bits of $x$ taken as a number, where the first (most significant) bit is considered the $0^{\text{th}}$. Equal to $(x \gg (z - b))\text{ }\%\text{ }(1 \ll (b-a))$. For example, $(0b100100)[2:5]$ on implied domain $[2^6]$ is $0b010 = 2$. Also, the default values are $a=0$ and $b=z$ if they are not specified.

Proof format

For a given $\text{plot\_seed} \in [2^{256}]$, a space parameter $33 \leq k \leq 59$, and a challenge $\text{Chall} \in [2^{256}]$ chosen by a verifier, a proof of space ${\Large\pi}$ is $64k$ bits:

${\Large\pi}_{\text{Chall; plot\_seed, k}} = x_1, x_2, ..., x_{64}$ (with $x_i \in [2^k]$) satisfying:

$\begin{aligned} \mathbf{M}(f_1(x_1), f_1(x_2)), \mathbf{M}(f_1(x_3), f_1(x_4)), \dots &\small\text{ [32 matches: } (x_1, x_2), (x_3, x_4), (x_5, x_6), \cdots]\\ \mathbf{M}(f_2(x_1, x_2), f_2(x_3, x_4)), \mathbf{M}(f_2(x_5, x_6), f_2(x_7, x_8)), \dots &\small\text{ [16 matches: } (x_1, \dots, x_4), (x_5, \dots, x_8), \cdots]\\ \mathbf{M}(f_3(x_1, x_2, x_3, x_4), f_3(x_5, x_6, x_7, x_8)), \dots &\small\text{ [8 matches: } (x_1, \dots, x_8), (x_9, \dots, x_{16}), \cdots]\\ \mathbf{M}(f_4(x_1, \dots, x_8), f_4(x_9, \dots, x_{16})), \dots &\small\text{ [4 matches: } (x_1, \dots, x_{16}), (x_{17}, \dots, x_{32}), \cdots]\\ \mathbf{M}(f_5(x_1, \dots, x_{16}), f_5(x_{17}, \dots, x_{31})), \dots &\small\text{ [2 matches: } (x_1, \dots, x_{32}), (x_{33}, \dots, x_{64})]\\ \mathbf{M}(f_6(x_1, \dots, x_{32}), f_6(x_{33}, \dots, x_{64})) &\small\text{ [1 match]}\\ \underset{k}{\text{trunc}}\big(\text{Chall} \big) = \underset{k}{\text{trunc}}\big(f_7(x_1, x_2, \dots, x_{64})\big) &\\ \end{aligned}$

(All the $\mathbf{M}$ matching functions return $\text{True}$.)

Here,

• $\mathbf{M}(x, y)$ is a matching function $\mathbf{M} : [2^{k+param\_EXT}] \times [2^{k+param\_EXT}] \rightarrow \{\text{True}, \text{False}\}$;
• $f$ is a high level function that composes $\mathcal{A}$ with $\mathcal{C}$;
• $\mathcal{A'}$ and $\mathcal{A_t}$ are high level hash functions that call $AES_{256}$ or $AES_{128}$ on their inputs in some way, using the plot seed as a key
• $\mathcal{C}$ is a collation function $\mathcal{C}_t : \underset{(2^{t-2} \text{times})}{[2^k] \times \cdots \times [2^k]} \rightarrow [2^{bk}]$ for $b = \text{colla\_size}(t)$
• Plot seed is an initial seed that determines the plot and proofs of space

Proof Quality String

The quality string (which can be used for blockchain consensus) for a valid proof ${\Large\pi}_{\text{Chall; plot\_seed, k}}$ is defined as $x_{2a+1} \mathbin{\|} x_{2a+2}$ where $a=Chall \% 32$, and the $x$ values are sorted in plot order. Plot ordering is a permutation of ${\Large\pi}$ $x$ values into ${\Large\pi}'$ $x'$ values such that:

$\text{for s in } [1, 2, 3, 4, 5] \text{ for i in } [0, \dots, 2^{6-s}-1], \text{ in order}$

$L = [x_{i*2^{s}+1}, \dots, x_{i*2^{s}+2^{s-1}}] , R= [x_{i*2^{s}+2^{s-1}+1}, \dots, x_{(i+1)*2^{s}}]$

$L$ and $R$ are switched, iff $L < R$, compared element-wise, from first to last.

Definition of parameters, and $\mathbf{M}, f, \mathcal{A}, \mathcal{C}$ functions:

Parameters:

We have the following parameters, precalculated values, and functions:

$\begin{aligned} \text{param\_EXT} &= 5\\ \text{param\_M} &= 1 \ll \text{param\_EXT} = 32\\ \text{param\_B} &= 60\\ \text{param\_C} &= 509\\ \text{param\_BC} &= \text{param\_B} * \text{param\_C}\\ \text{param\_c1} = \text{param\_c2} &= 10000\\ \text{bucket\_id}(x) &= \big\lfloor\frac{x}{\small\text{param\_BC}}\big\rfloor\\ \text{b\_id}(x), \text{c\_id}(x) &= \text{divmod}(x \% \text{param\_BC}, \text{param\_C}) \\ \text{colla\_size(t)} &= \begin{cases} 1 &\text{if } t = 2\\ 2 &\text{if } t = 3 \text{ or } t = 7\\ 3 &\text{if } t = 6\\ 4 &\text{if } t = 4 \text{ or } t = 5\\ \text{undefined} &\text{else} \end{cases} \end{aligned}$

$f$ functions:

For convenience of notation, we have functions $f_1, f_2, \cdots f_7$ with:

$\begin{aligned} f_t &: \underset{(2^{t-1} \text{times})}{[2^k] \times \cdots \times [2^k]} \rightarrow [2^{k + \small\text{param\_EXT}}] \\ f_1(x_1) &= \mathcal{A}'(x_1)\\ f_2(x_1, x_2) &= \mathcal{A}_2(\mathcal{C}_2(x_1), \mathcal{C}_2(x_2)) \oplus f_1(x_1)\\ f_3(x_1, x_2, x_3, x_4) &= \mathcal{A}_3(\mathcal{C}_3(x_1, x_2), \mathcal{C}_3(x_3, x_4)) \oplus f_2(x_1, x_2)\\ f_4(x_1, \dots, x_8) &= \mathcal{A}_4(\mathcal{C}_4(x_1, \dots, x_4), \mathcal{C}_4(x_5, \dots, x_8)) \oplus f_3(x_1, \dots, x_4)\\ f_5(x_1, \dots, x_{16}) &= \mathcal{A}_5(\mathcal{C}_5(x_1, \dots, x_8), \mathcal{C}_5(x_9, \dots, x_{16})) \oplus f_4(x_1, \dots, x_8)\\ f_6(x_1, \dots, x_{32}) &= \mathcal{A}_6(\mathcal{C}_6(x_1, \dots, x_{16}), \mathcal{C}_6(x_{17}, \dots, x_{32})) \oplus f_5(x_1, \dots, x_{16})\\ f_7(x_1, \dots, x_{64}) &= \mathcal{A}_7(\mathcal{C}_7(x_1, \dots, x_{32}), \mathcal{C}_7(x_{33}, \dots, x_{64})) \oplus f_6(x_1, \dots, x_{32})\\ \end{aligned}$

Matching function $\mathbf{M}$:

Then the matching function $\mathbf{M}$ is:

$\begin{aligned} \mathbf{M}(l, r) &: [2^{k+param\_EXT}] \times [2^{k+param\_EXT}] \rightarrow \{\text{True}, \text{False}\}\\ \\ \mathbf{M}(l, r) &= \begin{cases} \text{True} & \text{if } {\small \text{bucket\_id}(l) + 1 = \text{bucket\_id}(r)}\\ & \text{and } \exists m : 0 \leq m < \text{param\_M with:}\\ & \text{b\_id}(r) - \text{b\_id}(l) \equiv m \pmod{\text{param\_B}} \text{ and}\\ & \text{c\_id}(r) - \text{c\_id}(l) \equiv (2m + (\text{bucket\_id}(l) \% 2))^2 \pmod{\text{param\_C}}\\ \text{False} & \text{else} \end{cases} \end{aligned}$

$\large \mathcal{A'}$ function:

$\begin{aligned} \mathcal{A'}(x) = (AES_{256}(0, plot\_seed) \mathbin{\|} AES_{256}(1, plot\_seed) \mathbin{\|} ... )[kx : kx + k] \mathbin{\|} (x \text{1}\% \text{param\_M})& \\ \end{aligned}$

$\begin{aligned} \Rightarrow \mathcal{A}'(x) = \begin{cases} AES_{256}(q, plot\_seed)[r:r+k] \mathbin{\|} (x \text{1}\% \text{param\_M}) &\text{if } r+k \leq 128\\ AES_{256}(q, plot\_seed)[r:] \mathbin{\|} AES_{256}(q+1)[:r+k-128] \mathbin{\|} (x \text{1}\% \text{param\_M})&\text{else}\\ \end{cases} \end{aligned}$

for $(q, r) = \text{divmod}(x*k, 128)$.

$\large \mathcal{A_t}$ function:

$\begin{aligned} \mathcal{A}_t(x, y) := { \begin{cases} \underset{k+\text{param\_EXT}}{\text{trunc}}\big( A(x \mathbin{\|} y) \big) &\text{if } 0 \leq \text{size} \leq 128 \\ \underset{k+\text{param\_EXT}}{\text{trunc}}\big( A(A(x) \oplus y) \big) &\text{if } 129 \leq \text{size} \leq 256 \\ \underset{k+\text{param\_EXT}}{\text{trunc}}\big( A(A(x_{high}) \oplus A(y_{high}) \oplus A(x_{low} \mathbin{\|} y_{low})) \big) &\text{if } 257 \leq \text{size} \leq 384 \\ \underset{k+\text{param\_EXT}}{\text{trunc}}\big( A(A(A(x_{high}) \oplus x_{low}) \oplus A(y_{high}) \oplus y_{low}) \big) &\text{if } 385 \leq \text{size} \leq 512 \\ \end{cases} } \end{aligned}$

where

$\begin{aligned} A(c) &= AES_{128}(c, \underset{128}{\text{trunc}}(plot\_seed)) \\ x_{high}, x_{low} &= \text{divmod(}x\text{, 1 }\ll\text{ 128)}\\ y_{high}, y_{low} &= \text{divmod(}y\text{, 1 }\ll\text{ 128)}\\ \text{size} &= 2 * k * \text{colla\_size}(t)\\ \end{aligned}$

Collation function $\mathcal{C}$:

$\begin{aligned} \mathcal{C}_{t}(x_1, ..., x_{2^{t-2}}) := { \begin{cases} x_1 &\text{if } t = 2\\ x_1 \mathbin{\|} x_2 &\text{if } t = 3\\ x_1 \mathbin{\|} x_2 \mathbin{\|} x_3 \mathbin{\|} x_4 &\text{if } t = 4\\ (x_1 \mathbin{\|} \dots \mathbin{\|} x_4) \oplus (x_5 \mathbin{\|} \dots \mathbin{\|} x_8) &\text{if } t = 5\\ \underset{3k}{\text{trunc}}\big( (x_1 \mathbin{\|} \dots \mathbin{\|} x_4) \oplus (x_5 \mathbin{\|} \dots \mathbin{\|} x_8) \oplus (x_9 \mathbin{\|} \dots \mathbin{\|} x_{12}) \oplus (x_{13} \mathbin{\|} \dots \mathbin{\|} x_{16}) \big) &\text{if } t = 6\\ = (x_1 \mathbin{\|} x_2 \mathbin{\|} x_3) \oplus (x_5 \mathbin{\|} x_6 \mathbin{\|} x_7) \oplus (x_9 \mathbin{\|} x_{10} \mathbin{\|} x_{11}) \oplus (x_{13} \mathbin{\|} x_{14} \mathbin{\|} x_{15}) \\ \underset{2k}{\text{trunc}}\big( (x_1 \mathbin{\|} \dots \mathbin{\|} x_4) \oplus \cdots \oplus (x_{29} \mathbin{\|} \dots \mathbin{\|} x_{32}) \big) &\text{if } t = 7\\ = (x_1 \mathbin{\|} x_2) \oplus (x_5 \mathbin{\|} x_6) \oplus \cdots \oplus (x_{29} \mathbin{\|} x_{30}) \end{cases} } \end{aligned}$

How do we implement Proof of Space?

Proof of space is composed of three algorithms: plotting, proving and verication.

Plotting

Plotting Tables (Concepts)

Plotting is our term for the method of writing data to disk such that we can quickly retrieve 1 or more proofs (if they exist) for a given challenge. The plot refers to the contents of the disk. While proof retrieval must be fast, the plotting process can take time, as it is only done once, by the prover. In the Chia Blockchain, these provers are referred to as farmers, since they create and maintain the plots. That is, a farmer creates and efficiently stores data on disk, for a given plot seed, that allows them to find proofs $\Large{\Pi}$ that meet the above requirements. We first discuss general concepts related to the problem of plotting.

Tables

There are 7 tables, each with $O(2^k)$ entries, where $k$ is the space parameter. Each table has information for finding $x_i$'s that fulfill the matching condition of that table and all previous tables.

For convenience, we will denote the new matching condition introduced at each table: $\mathbf{M_1}(x_1, x_2) = \mathbf{M}(f_1(x_1), f_1(x_2))$ for the first table, $\mathbf{M}_2(x_1, x_2, x_3, x_4) = \mathbf{M}(f_2(x_1, x_2), f_2(x_3, x_4))$ for the second table, and so on.

We will also refer to the total matching conditions of a table and all previous tables:

$\widetilde{\mathbf{M}}_t(x_1, \dots, x_{2^t}) = \widetilde{\mathbf{M}}_{t-1}(x_1, \dots, x_{2^{t-1}}) \land \widetilde{\mathbf{M}}_{t-1}(x_{2^{t-1} + 1}, \dots, x_{2^{t}}) \land \mathbf{M}_t(x_1, \dots, x_{2^t})$

Tables conceptually are broken down into a sequence of entries that represent $x_i$ tuples that satisfy these total matching conditions. For example, each entry of the third table has information on recovering $(x_1, \dots, x_8)$ tuples such that $\widetilde{\mathbf{M}}_3(x_1, \dots, x_{8})$ is $\text{True}$, ie. $\mathbf{M}_3(x_1, \dots, x_8), \mathbf{M}_2(x_1, \dots, x_4), \mathbf{M}_2(x_5, \dots, x_8), \mathbf{M}_1(x_1, x_2), \cdots, \mathbf{M}_1(x_7, x_8)$ are all $\text{True}$.

We also refer to the left and right halves of $x_i$ tuples: if we have an entry representing tuple $(x_1, \dots, x_{2^t})$, then the halves are $\text{left\_half} = (x_1, \dots, x_{2^{t-1}})$ and $\text{right\_half} = (x_{2^{t-1} + 1}, \dots, x_{2^t})$.

These entries can also be ordered, so that (relative to some ordering $\sigma$):

$\text{Table}_{t; \sigma} = E_{\{t, 0\}}, E_{\{t, 1\}}, \cdots, E_{\{t, N_t - 1\}}$

where $N_t$ is the number of entries in the $t^{\text{th}}$ table.

The final table, $T_7$, is special, in that it does not contain information for a matching condition, but instead stores $f_7$ bits, which we will refer to as the final outputs of the plot. The farmer must be able to look up final outputs efficiently, using the challenge, and the data format used to achieve this is described in Checkpoints.

Table Positions

For each entry, instead of writing the $x_i$ data, we refer to the position of that entry in the previous table.

For example, in the third table, each entry conceptually is $E_{\{3, i\}} = (\text{left\_half}: (x_1, x_2, x_3, x_4), \text{right\_half}: (x_5, x_6, x_7, x_8))$, but we can store it instead as $E_{\{3, i\}} = (\text{left\_half}: E_{\{2, j_1\}}, \text{right\_half}: E_{\{2, j_2\}})$ for some $j_1, j_2$.

This makes it clear that (given we have stored all previous tables), storing $j_1, j_2 \in N_{t-1}$ is sufficient to recover $E_{\{t, i\}}$.

Compressing Entry Data

We have points (positions to the previous table) $(j_1, j_2) \in N_t$ that we want to compress. Actually, because we can recover the order of these indices (by checking $\mathbf{M}(\dots)$ later), we only need to store the set of them: ie., store $(\max(j_1, j_2), \min(j_1, j_2))$ as data from a discrete uniform distribution (as we will see) in the "triangle space" $\text{Triangle}_t = \{(x,y) | x, y \in [N_t], x > y\}$. Since $x$ is always greater, you can imagine these entries as random points in a two dimensional $N_t$ by $N_t$ triangle.

We can exhibit a bijection from this space to a line $\text{Line}_t = \Big[\frac{(N_t - 1)(N_t - 2)}{2}\Big]$:

$\begin{aligned} (x, y) \in \text{Triangle}_t \Rightarrow& \frac{x(x-1)}{2} + y \in \text{Line}_t\\ p \in \text{Line}_t \Rightarrow& \Big(x, p - \frac{x(x-1)}{2}\Big) \in \text{Triangle}_t, \text{where } x = \Big\lfloor \frac{\sqrt{8p + 1} + 1}{2} \Big\rfloor\\ \end{aligned}$

(Note: this data is not actually in a discrete uniform distribution (eg. no line points are repeated), though we will see that this has a negligible effect.)

We can compress these points in $\text{Line}_t$, referred to as $line\_points$, by converting them to delta format, then encoding them with ANS.

Delta Format

Suppose we have a set of $K$ points from a discrete uniform distribution: $x_1, x_2, \dots, x_K$ chosen independently from $[RK]$. (In our specific case, we have $N_t$ points from $\text{Line}_t$). Here, $R$ is a "space to entries ratio" parameter.

We can sort these points and store only the deltas: $\delta_i = x_{i-1} - x_{i}$ for $i = 1, \dots, K$ (and $x_0 = 0$). It is clear that from these deltas, we can recover the set of points. We say that the points are in delta format if we are storing these deltas instead of points.

ANS Encoding of Delta Formatted Points

We can encode these deltas (from points in a discrete uniform distribution) using an Asymmetric Numeral System (ANS [4]) encoding, an encoding scheme that is essentially lossless and offers extremely fast compression and decompression speeds for fixed distributions. A requirement to do so is that we know the distribution of $\Delta_i$. From mathematics pertaining to order statitics, it can be proved that the pdf (probability density function) of $\delta$ is independent of $i$, and equal to:

$P(\Delta_i = X; R) = \begin{cases} 1 - (\frac{e-1}{e})^{\frac{1}{R}} & \text{if }X = 0\\ (e^{\frac{1}{R}} - 1)(e-1)^{\frac{1}{R}}(e^{\frac{-(X+1)}{R}})& \text{else}\\ \end{cases}$

Stub and Small Deltas

In the previous discussion, the $\Delta_i$'s are approximately $\Delta_i \approx N_t \approx 2^k$ for line points of size $2^{2k}$, makes certain compression and decompression operations infeasible. To mitigate this, we convert $\Delta_i$'s to "stub and small delta" form:

$(\delta_i, \text{stub}_i) = \text{divmod}(\Delta_i, 1 \ll (k-2))$

After, we can write the stubs directly, and an encoding of the $\delta_i$'s. The distributions of $\text{stub}_i \in [2^{k-2}]$ and $\delta_i \in [(N_t - 1)(N_t - 2) \gg (k-1)]$ are approximated (to a negligible error) by the discrete uniform distribution on these domains, which is important for our use of ANS.

We can also calculate the appropriate space to entries ratio parameter for the current table:

$R_t = \frac{(N_t-1)(N_t-2)}{N_t * 2^{k-1}} \approx 2\Big(\frac{N_t}{2^k}\Big)$

Parks

For each table $t$, we often have the need to efficiently find the $i$-th entry in that table: an element of $\text{Line}_t$.

Our strategy is to group these entries into parks: $\text{param\_EPP} = 2048$ entries per park (EPP), so that:

$\text{Park}_i \text{ encodes } E_{\{t, i * \text{param\_EPP}\}}, E_{\{t, i * \text{param\_EPP} + 1\}}, \dots, E_{\{t, i * \text{param\_EPP} + \text{param\_EPP} - 1\}}$

Each park encodes information about its entries ($e_0, e_1, \dots$) as follows:

• The value of the first entry $e_0 \in \text{Line}_t$;
• The $(\text{param\_EPP} - 1)$ $\Delta$'s:
  • The $(\text{param\_EPP} - 1)$ $\text{stub}$'s, written directly [$k-2$ bits per stub];
  • The $(\text{param\_EPP} - 1)$ $\delta$'s, ANS encoded;

Then, to find the $i^\text{th}$ entry in the park, we can find the $r^\text{th}$ entry in $\text{Park}_q$, where $(q, r) = \text{divmod}(i, \text{param\_EPP})$.

One wrinkle with this system is that it is highly desired that each park starts at a fixed location on disk. One simple way to achieve this is to allocate more than enough space to store the ANS encoding of the $\delta$'s. Because the variance of $H(\delta)$ is very low as a percentage of the total disk space of the park (where $H$ represents Shannon entropy), this is a reasonable strategy, as the wasted space is very small.

We can bound the probability that the parks have sufficient space to store these entries, by calculating the mean and standard deviation of the information content in an entire park:

$\begin{aligned} \mu &= \sum_{X \in P(X; R)} -P(X) \log_2 P(X) \\ \sigma &= \sqrt{\sum_{X \in P(X; R)} P(X) ((-\log_2 P(X)) - \mu)^2} \\ \hat{\mu} &= \mu * (\text{\small param\_EPP} - 1) \\ \hat{\sigma} &= \sigma * \sqrt{\text{\small param\_EPP} - 1} \\ \end{aligned}$

For eg. $R = 1.0$ and $\text{param\_EPP} = 2048$, we have $\hat{\mu} = 3885.33, \hat{\sigma} = 52.21$. Because $\text{InverseCDF}[\text{NormalDistribution}[0,1], (99.9\%)^{2^{-40}}] < 8.0$, we can choose a park size of $\hat{\mu} + 8 \hat{\sigma}$ to have an approximately 99.9% chance to store a single table of $2^{40}$ entries.

More complicated systems of writing parks to disk are possible that offer minor compression improvements, based on recovering the endpoints of parks that have been coalesced. For a more thorough discussion, see Variable Sized Parks.

Checkpoint Tables

Retriving a proof of space for a challenge, as explained in Proving, requires efficienty looking up a final output $f_7$ in the final table. Therefore, in $\text{Table}_7$ entries are sorted by $f_7$. In order to compress the on disk representation, we can store deltas between each $f_7$, which are small, since there are approximately $2^k$ uniformly distributed integers of size $k$.

We can efficiently encode these deltas with a variable scheme like Huffman encoding. For simplicity, we allocate a fixed amount of space for encoding deltas for $param\_c_1 = 10000$ entries, and allow 3 bits per delta to prevent overflows, storing them in parks.

However, every $param\_c_1$ entries, the $f_7$ will drift away from the position in the table, for example entry 10,000 might be stored in position 10,245, so we store a checkpoint every $param\_c_1$ entries, in a new table, $C_1$. Furthermore, since these checkpoints do not fit in memory for a large $k$, we can store an additional checkpoint every $param\_c_2$ $C_1$ entries, in yet a new table, $C_2$. Since there will be approximately $\frac{2^k}{c_{1}*c_{2}}$ $C_2$ entries, we can store these in memory.

Plotting Algorithm

Final Disk Format

Let $pos_{t;L}$ and $pos_{t;R}$ denote the positions in $Table_t$ of a $\text{Table}_{t+1}$ entry. That is, a $Table_{t+1}$ entry is composed of two indeces into $Table_t$.

$f_7$ is a final output of function $f_7$, which is compared to the challenge at verification. The $N$ values (number of entries in a table) are calculated in the Space Required section. The final format is composed of a table that stores the original $x$ values, 5 tables that store positions to the previous table, a table for the final outputs, and 3 small checkpoint tables.

The final disk format is the following:

Full algorithm

The plotting algorithm $\text{PlotAlgo1}$ takes as input a unique $\text{plot\_seed} \in [2^{256}]$, a space parameter $33 \leq k \leq 60$, a param object $param$, and deterministically outputs a final file $F$.

$Sort$, or sort on disk, takes in a table on disk, and performs a full ascending sort, starting at a specified bit position.

# 1 Forward propagation phase
For x in 0...2^k - 1:
    Compute f1(x)
    Write (f1(x), x) to table1

For table in 1..6:
      Sort tablei by (fi(x), pos, offset). for table1, by f1(x)
      For entry in tablei:
            if entries L and R match:
                  Compute fi+1(CL, CR)  for table1, M=x
                  C = Collation_i(CL, CR)
                  Store (fi+1, pos, offset, C) in tablei+1

# 2 Backpropagation phase
For table in 6..1:
      Iterate through tablei and tablei+1:
           Drop unused entries in tablei
           sort_key = table == 7 ? fi : tablei+1pos
           Rewrite used entries in tablei as (sort_key, pos, offset)
           Rewrite entries in tablei+1 as (sort_key, pos, offset)
      If i > 1:
           Sort tablei by (pos, offset)

# 3 Compression phase
For table in 1..6:
      Iterate through tablei and tablei+1:
           Read sort_key, pos, offset from tablei+1
           Read tablei entries in that pos and offset:  eL, eR
           y, x = sort(eL.newPos, eR.newPos)
           line_point = x*(x-1)//2 + y
           Rewrite tablei+1 entry as (line_point, sort_key)
      Sort tablei+1 by line_point
      For entry e, i in enumerate(tablei+1):
           newPos = i
           Rewrite e as (sort_key, newPos)
           Write compressed e.line_point deltas to table Pi
      Sort tablei+1 by sort_key

# 4 Checkpoints phase
For entries in table7:
      Compress f7 entries into C1,C2,C3 tables
      Write pos6 entries to table P7k

Phase 1: Forward Propagation

The purpose of the forward propagation phase is to compute all tables where the matching conditions are met, and to compute all final outputs $f_7$. For each $Table_i$, we compute the correspoding $f$ function on the $x$ values (actually on the collated values), and write the outputs to disk to $Table_{i+1}$, along with the corresponding positions in $Table_i$, and the collated $x$ values.

Then, $Sort$ is performed on the new table, to sort it by output $f_i$. This allows us to check for the matching condition efficiently, since matching entries will have adjacent $bucket\_id$s. Matches can be found by using a sliding window which reads entries from disk, one group at a time. $C_3$ to $C_7$ refers to the output of the collation functiod for each n

File $T$ will store the following data after phase 1:

Pointers in Phase 1 and 2 are stored in pos, offset format. Pos offset format represents a match, by storing a $k+1$ bit position, and a much smaller 9 or 10 bit offset.

Duting phase 1, entries are close to their matches, so storing the offset is efficient. This will change into double pointer format, which stores two $k$ bit pointers, but uses sorting and compression to reduce each entry: to a size much closer to $k$.

Phase 2: Backpropagation

Note that after phase 1, the temporary file is sufficient to create proofs of space. $f_7$ can be looked up, and the tables can be followed backwards until the $x$ values in $Table_1$ are reached. However, this is not space efficient.

The purpose of the backpropagation step is to remove data which is not useful for finding proofs. During this phase, all entries which do not form part of the matching condition in the next table, are dropped. Positions in the next table are adjusted to account for the dropped entries. The Collation outputs ($C$ values) are also dropped, since they are only useful for forward propagation. For a more efficient compression phase, each entry is given a $sort\_key$, which corresponds to it's position in the table. This is more efficient than storing $(f_t, pos_{t-1}, offset)$ to represent it's position.

Implementation of this phase requires iterating through two tables at once, a left and a right table, checking which left entries are used in the right table, and dropping the ones that are not. Finally, a sort by position is required before looking at the right table, so it's possible to read a window into both tables at once, in a cache-local manner, without reading random locations on disk.

File $T$ will store the following data after phase 2:

Phase 3: Compression

The purpose of phase 3 is to convert from the (pos, offset) format, which requires tables to be sorted according to their $bucket\_ids$ for retrieval, to a double pointer format, which allows storing entries in a more compressed way, and sorts by $line\_point$. In the pos offset format around k bits are required for the pos and 8 bits for the offset, for a total of around $k+8$ bits per entry, whereas in double pointer format, we can average around 2 bits per entry overhead.

With the compressed format, two pointers can be stored in around $k+2$ bits, using the strategies described in Compressing Entry Data.

The goal for the double pointer format is to store two $k$ bit pointers to a previous table ($2k$ bits of information), as compactly as possible. This is achieved by having each entry $E_{t, i}$ be represented as pointers to the previous table, where the previous table is sorted by $line\_point$. This makes positions random (i.e. it’s no longer sorted by matches/$bucket\_id$). Then we take these $2k$ bit $line\_points$, and encode the deltas between each one, each which will be around $k$ bits. This phase requires some careful sorting and mapping from old positions to new positions, since we don't have as much memory as disk space.

As we iterate from $Table_1$ to $Table_7$, the compression is performed, and each compressed table is written to the final file $F$.

File $T$ will store the following data after phase 3: