New SSZ type to represent a flexible container with stable serialization and merkleization
|Authors||Etan Kissling (@etan-status)|
Table of Contents
- Backwards Compatibility
- Test Cases
- Reference Implementation
- Security Considerations
This EIP introduces a new Simple Serialize (SSZ) type to represent
StableContainer[N] is an SSZ
Container with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.
Stable containers are currently not representable in SSZ. Adding support provides these benefits:
Stable signatures: Signing roots derived from a
StableContainer[N]never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.
Stable merkle proofs: Merkle proof verifiers that check specific fields of a
StableContainer[N]do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same generalized indices.
Compact serialization: SSZ serialization is compact; inactive fields do not consume space.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119 and RFC 8174.
Similar to the regular SSZ
StableContainer[N] defines an ordered heterogeneous collection of fields.
N indicates the potential maximum number of fields to which it can ever grow in the future.
N MUST be
As part of a
StableContainer[N], fields of type
Optional[T] MAY be defined. Such fields can either represent a present value of SSZ type
T, or indicate absence of a value (indicated by
None). The default value of an
class Example(StableContainer): a: uint64 b: Optional[uint32] c: uint16
For the purpose of serialization,
StableContainer[N] is always considered “variable-size” regardless of the individual field types.
The serialization and merkleization of a
StableContainer[N] remains stable as long as:
- The maximum capacity
Ndoes not change
- The order of fields does not change
- New fields are always added to the end
- Required fields remain required
T, or become an
- Optional fields remain
Optional[T], or become a required
When an optional field becomes required, existing messages still have stable serialization and merkleization, but will be rejected on deserialization if not present.
StableContainer[N] is defined similarly to the existing logic for
Container. Notable changes are:
Bitvector[N]is constructed, indicating active fields within the
StableContainer[N]. For required fields
Tand optional fields
Optional[T]with a present value (not
Truebit is included. For optional fields
Falsebit is included. The
Bitvector[N]is padded with
Falsebits up through length
- Only active fields are serialized, i.e., fields with a corresponding
Truebit in the
- The serialization of the
Bitvector[N]is prepended to the serialized active fields
- If variable-length fields are serialized, their offsets are relative to the start of serialized active fields, after the
def is_active_field(element): return not is_optional(element) or element is not None # Determine active fields active_fields = Bitvector[N](([is_active_field(element) for element in value] + [False] * N)[:N]) active_values = [element for element in value if is_active_field(element)] # Recursively serialize fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in active_values] variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in active_values] # Compute and check lengths fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts] variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Interleave offsets of variable-size parts with fixed-size parts variable_offsets = [serialize(uint32(sum(fixed_lengths + variable_lengths[:i]))) for i in range(len(active_values))] fixed_parts = [part if part != None else variable_offsets[i] for i, part in enumerate(fixed_parts)] # Return the concatenation of the active fields `Bitvector` with the active # fixed-size parts (offsets interleaved) and the active variable-size parts return serialize(active_fields) + b"".join(fixed_parts + variable_parts)
Deserialization of a
StableContainer[N] starts by deserializing a
Bitvector[N]. That value MUST be validated:
- For each required field, the corresponding bit in the
- For each optional field, the corresponding bit in the
Bitvector[N]is not restricted
- All extra bits in the
Bitvector[N]that exceed the number of fields MUST be
The rest of the data is deserialized same as a regular SSZ
Container, consulting the
Bitvector[N] to determine what optional fields are present in the data. Absent fields are skipped during deserialization and assigned
The merkleization specification is extended with the following helper functions:
chunk_count(type): calculate the amount of leafs for merkleization of the type.
N, regardless of the actual number of fields in the type definition
mix_in_aux: Given a Merkle root
rootand an auxiliary SSZ object root
hash(root + aux).
To merkleize a
Bitvector[N] is constructed, indicating active fields within the
StableContainer[N], using the same process as during serialization.
hash_tree_root(value) of an object
value is extended with:
mix_in_aux(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), hash_tree_root(value.active_fields))if
Current SSZ types are only stable within one version of a specification, i.e., one fork of Ethereum. This is alright for messages pertaining to a specific fork, such as attestations or beacon blocks. However, it is a limitation for messages that are expected to remain valid across forks, such as transactions or receipts. In order to support evolving the features of such perpetually valid message types, a new SSZ scheme needs to be defined.
To avoid restricting design space, the scheme has to support extension with new fields, obsolescence of old fields, and new combinations of existing fields. When such adjustments occur, old messages must still deserialize correctly and must retain their original Merkle root.
Typically, the individual
Union cases share some form of thematic overlap, sharing certain fields with each other. In a
Union, shared fields are not necessarily merkleized at the same generalized indices. Therefore, Merkle proof systems would have to be updated each time that a new flavor is introduced, even when the actual changes are not of interest to the particular system.
Furthermore, SSZ Union types are currently not used in any final Ethereum specification and do not have a finalized design themselves. The
StableContainer[N] serializes very similar to current
Union[T, U, V] proposals, with the difference being a
Bitvector[N] as a prefix instead of a selector byte. This means that the serialized byte lengths are comparable.
Optional[T] is modeled as an SSZ type, each individual field introduces serialization and merkleization overhead. As an
Optional[T] would be required to be “variable-size”, lots of additional offset bytes would have to be used in the serialization. For merkleization, each individual
Optional[T] would require mixing in a bit to indicate presence or absence of the value.
Additionally, every time that the number of fields reaches a new power of 2, the Merkle roots break, as the number of chunks doubles. The
StableContainer[N] solves this by artificially extending the Merkle tree to
N chunks regardless of the actual number of fields currently specified. Because
N is constant across specification versions, the Merkle tree shape remains stable. The overhead of the additional empty placeholder leaves only affects serialization of the
Bitvector[N] (1 byte per 8 leaves); the number of required hashes during merkleization only grows logarithmically with
StableContainer[N] is a new SSZ type and does not conflict with other SSZ types currently in use.
See EIP assets.
See EIP assets, based on
Copyright and related rights waived via CC0.
Please cite this document as:
Etan Kissling (@etan-status), "EIP-7495: SSZ StableContainer [DRAFT]," Ethereum Improvement Proposals, no. 7495, August 2023. [Online serial]. Available: https://eips.ethereum.org/EIPS/eip-7495.