A high performance Kaspa PostgreSQL indexer implemented in Rust.
50K+
A high performance Kaspa PostgreSQL indexer implemented in Rust.
The indexer has been implemented from scratch by deriving the functional spec of kaspa-db-filler.
As part of this process the database schema was reworked to better support concurrency.
This means that databases populated by the lAmeR1/kaspa-db-filler must be migrated to be compatible.
A schema migration script has been developed and is available here.
A compatible version of the kaspa-rest-server is available here.
To maximize the performance for your specific needs you should take care to disable any table you don't need through command line flags.
See --help for a list of optional fields.
In addition to optional tables, many fields can be left empty if they are not required for your use case.
Use exclude-fields arguments to fine tune. See --help for a list of optional fields.
If addresses_transactions_table is NOT disabled and exclude-fields contains tx_out_script_public_key_address (and not tx_out_script_public_key),
the indexer will use scripts_transactions instead of addresses_transactions for indexing addresses for > 25% space savings.
Make sure to experiment with the --batch-scale parameter, as it greatly affects insert performance.
You should also tune Postgres configuration to your specific hardware, here is an example for a server with 12GB RAM and SSD storage:
shared_buffers = 2GB
work_mem = 128MB
effective_io_concurrency = 32
checkpoint_timeout = 5min
max_wal_size = 4GB
min_wal_size = 80MB
effective_cache_size = 8GB
In addition, I highly recommend running Postgres on ZFS with compression=lz4 (or zstd) for space savings as well as for improving performance. Make sure to also set recordsize=16k.
The indexer is able to keep up with the 10bps testnet (TN11) under full load (2000+tps) as long as Postgres is running on a sufficiently high-end NVMe.
By disabling optional tables and fields you can bring the requirements down if running on lesser hardware.
The indexer will begin collecting data from the point in time when it's started.
If you have an archival node, you can specify the start-block using the --ignore_checkpoint argument and specify an older start block.
Please make contact with us on the Kaspa Discord if you need a pg_dump-file of historical records.
MIT, which means this software can be freely modified to any specific need and redistributed (under certain terms).
Please be so kind as to contribute back features you think could be beneficial to the general community.
kaspa:qrjtsnnpjyvlmkffdqyayrny3qyen9yjkpuw7xvhsz36n69wmrfdyf3nwv67t
The default is well suited for a block explorers and the like, which is overkill for many other use cases.
This is the recommended starting point for exchanges and other 'light' integrators:
disable:
block_parent_table,
blocks_transactions_table,
addresses_transactions_table,
rejected_transactions
exclude-fields:
block_accepted_id_merkle_root,
block_merge_set_blues_hashes,
block_merge_set_reds_hashes,
block_selected_parent_hash,
block_bits,
block_blue_work,
block_daa_score,
block_hash_merkle_root,
block_nonce,
block_pruning_point,
block_utxo_commitment,
block_version,
tx_hash,
tx_mass,
tx_payload,
tx_in_signature_script,
tx_in_sig_op_count,
tx_out_script_public_key_address
Example command arguments:
-u -s ws://your-kaspad:17110 -d postgres://postgres:postgres@your-db:5432 -l 0.0.0.0:8500 \
--prune-db --retention=7d \
--disable=block_parent_table,blocks_transactions_table,addresses_transactions_table,rejected_transactions \
--exclude-fields=block_accepted_id_merkle_root,block_merge_set_blues_hashes,block_merge_set_reds_hashes,block_selected_parent_hash,block_bits,block_blue_work,block_daa_score,block_hash_merkle_root,block_nonce,block_pruning_point,block_utxo_commitment,block_version,tx_hash,tx_mass,tx_payload,tx_in_signature_script,tx_in_sig_op_count,tx_out_script_public_key_address
docker run --rm -it supertypo/simply-kaspa-indexer -s ws://<kaspad_host>:17110 -d postgres://postgres:postgres@<postgres_host>:5432/postgres --exclude-fields=tx_out_script_public_key_address --prune-db --retention=7d
Make sure to adjust (or remove) pruning.
Paste the content below into docker-compose.yaml, and run docker compose up -d in the same folder.
The explorer will be available on http://localhost:8080 soon afterwards (expect a couple of hours for kaspad to fully sync though).
services:
kaspa_explorer:
container_name: kaspa_explorer
image: supertypo/kaspa-explorer:latest
restart: unless-stopped
ports:
- "127.0.0.1:8080:8080"
environment:
API_SERVER: http://localhost:8000
SOCKET_SERVER: ws://localhost:8001
ADDRESS_PREFIX: kaspa:
KASPA_UNIT: KAS
SUFFIX: DEV
kaspa_rest_server:
container_name: kaspa_rest_server
image: kaspanet/kaspa-rest-server:latest
restart: unless-stopped
environment:
KASPAD_WRPC_URL: ws://kaspad:17110
SQL_URI: postgresql+psycopg://postgres:postgres@kaspa_db:5432/postgres
USE_SCRIPT_FOR_ADDRESS: true
ports:
- "127.0.0.1:8000:8000"
simply_kaspa_socket_server:
container_name: simply_kaspa_socket_server
image: supertypo/simply-kaspa-socket-server:unstable
restart: unless-stopped
ports:
- "127.0.0.1:8001:8000"
command: -x 20 -s ws://kaspad:17110
simply_kaspa_indexer:
container_name: simply_kaspa_indexer
image: supertypo/simply-kaspa-indexer:latest
restart: unless-stopped
ports:
- "127.0.0.1:8500:8500"
command: -u -s ws://kaspad:17110 -d postgresql://postgres:postgres@kaspa_db:5432/postgres --exclude-fields=tx_out_script_public_key_address --prune-db --retention=7d
kaspad:
container_name: kaspad
image: supertypo/rusty-kaspad:latest
restart: unless-stopped
ports:
- "127.0.0.1:16110:16110"
- "127.0.0.1:17110:17110"
volumes:
- /var/kaspad:/app/data/
command: kaspad --yes --nologfiles --disable-upnp --utxoindex --rpclisten=0.0.0.0:16110 --rpclisten-borsh=0.0.0.0:17110
kaspa_db:
container_name: kaspa_db
image: postgres:17-alpine
restart: unless-stopped
shm_size: 4G
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- "127.0.0.1:5432:5432"
volumes:
- /var/kaspa_db/:/var/lib/postgresql/data/
Content type
Image
Digest
sha256:f28478e04…
Size
15.3 MB
Last updated
3 months ago
docker pull supertypo/simply-kaspa-indexer