Barre/compact_log: RFC6962 certificate transparency log with LSM-tree based storage

⚠️ This is a work in progress. While the CT functionality works, this should not be yet used in production.

A Certificate Transparency (CT) log implementation. CompactLog implements the RFC 6962 Certificate Transparency API on top of SlateDB to explore how LSM-tree storage can address traditional CT log scalability challenges.

This implementation provides a complete Certificate Transparency log that:

• Accepts X.509 certificate chains and pre-certificates
• Issues Signed Certificate Timestamps (SCTs)
• Maintains a cryptographically verifiable Merkle tree
• Provides inclusion and consistency proofs
• Stores data in cloud object storage (S3, Azure Blob) or local filesystem

CompactLog makes three fundamental design choices that differentiate it from other CT implementations:

1. LSM-tree storage via SlateDB instead of relational databases or custom storage engines.
2. STH-boundary versioning – only persisting tree state at published checkpoints.
3. Synchronous tree updates – achieving a Maximum Merge Delay (MMD) of 0 seconds.

Many CT log implementations have a Merge Delay (MMD) of minutes to hours, where submitted certificates aren’t yet included in the Merkle tree. This exists because:

• Many implementations issue SCTs immediately, then incorporate certificates later via background processes.
• Some implementations have expensive tree update operations.
• Consistency requires coordinating distributed components.

CompactLog eliminates a MD by reversing this order – certificates are incorporated before SCTs are issued:

Submission 1 ─┐
Submission 2 ─┼─ Wait up to 500ms ─→ Batch tree update ─→ All SCTs returned
Submission 3 ─┘                             └── Certificates already incorporated

The 500ms delay is submission latency, not a merge delay. Once SCTs are issued, certificates are already in the tree.

The batching system:

• Collects submissions for up to 500ms (configurable) to form a batch
• Updates the Merkle tree once for the entire batch
• Returns SCTs only after certificates are incorporated in the tree
• No background processing – certificates are immediately available for proofs

Traditional CT implementations:

Submit cert → Issue SCT immediately → [MMD period] → Incorporate in tree

CompactLog:

Submit cert → [Batch delay ≤500ms] → Incorporate in tree → Issue SCT

Result: Traditional logs have MMD measured in minutes/hours; CompactLog has an MMD of 0 seconds.

CompactLog versions nodes only at STH publication boundaries:

• Update nodes in-memory during batch operations
• Store O(log n) versioned nodes only at STH publication
• With STHs every k certificates: reduces versioned storage from O(n log n) to O(n log n / k)

Example: Publishing STHs every 1000 certificates reduces versioned storage overhead by 1000x.

# Core data
leaf:{index} → certificate/precert data
vnode:{node}@{version} → node hash (version = STH boundary)
nver:{node} → latest version of node

# Operational state
meta → current tree size
committed_size → last STH boundary
hash:{leaf_hash} → tree index
cert_sct:{cert_hash} → SCT data

# Certificate storage (deduplication)
cert:{cert_hash} → certificate binary data
entry:{index} → deduplicated log entry

CompactLog stores certificate chains using content-addressable storage:

1. Entry structure: Each log entry stores SHA-256 hashes of certificates rather than the certificates themselves
2. Certificate store: Certificates are stored separately under cert:{hash} keys
3. Deduplication: Multiple entries referencing the same certificate (e.g., intermediate CA certs) share the same stored copy
4. Reconstruction: The API reconstructs full certificate chains by resolving hash references during retrieval

The DeduplicatedLogEntry structure contains:

• Certificate hash (32 bytes)
• Chain certificate hashes (array of 32-byte hashes)
• Original metadata (timestamp, index, entry type)

Every operation maintains strict consistency:

• Reads see the latest committed STH state
• Writes are serialized through asynchronous locking (synchronous from client perspective)
• Proofs only available at STH boundaries (ensuring stable references)
• No eventual consistency – all operations are immediately visible

Create Config.toml or let the system generate defaults:

[server]
bind_addr = "0.0.0.0:8080"

[storage]
provider = "local"  # "aws", "azure", or "local"

[storage.local]
path = "/tmp/ct-log-storage"

[keys]
private_key_path = "keys/private_key.pem"
public_key_path = "keys/public_key.pem"

For cloud storage, configure provider-specific credentials in the respective sections.

# Start with default local configuration
cargo run --release

# With debug logging
RUST_LOG=debug cargo run --release

The system automatically generates ECDSA P-256 keys and default configuration if not present.

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!