Unverified Commit eba463c9 authored by kenshin samourai's avatar kenshin samourai Committed by GitHub
Browse files

Merge pull request #2 from kenshin-samourai/addresses_indexer

code cut
parents 608b2527 7cbb9a6d
version: 2
jobs:
build:
docker:
- image: rust:1.34.0-slim
steps:
- checkout
- run:
name: Install Dependencies
command: |
apt-get update
apt-get install -y clang cmake libsnappy-dev
rustup component add rustfmt
rustup component add clippy
- run:
name: Format
command: cargo fmt --all -- --check
- run:
name: Check
command: cargo check --all
- run:
name: Clippy
command: cargo clippy --all
- run:
name: Build
command: cargo build --all
- run:
name: Test
command: cargo test --all
target/
.git/
_*/
...@@ -5,3 +5,4 @@ _*/ ...@@ -5,3 +5,4 @@ _*/
*.sublime* *.sublime*
*~ *~
*.pyc *.pyc
Dockerfile
language: rust
rust:
- stable
cache: cargo
before_script:
- rustup component add rustfmt-preview
- rustup component add clippy
script:
- cargo fmt --all -- --check
- cargo check --all
- cargo clippy --all
- cargo build --all
- cargo test --all
This diff is collapsed.
[package] [package]
name = "electrs" name = "addrindexrs"
version = "0.8.0" version = "0.1.0"
authors = ["Roman Zeyde <me@romanzey.de>"] authors = ["Roman Zeyde <me@romanzey.de>", "kenshin-samourai <kenshin_samourai@tutanota.com>"]
description = "An efficient re-implementation of Electrum Server in Rust" description = "An efficient address indexer in Rust"
license = "MIT" license = "MIT"
homepage = "https://github.com/romanz/electrs" homepage = "https://github.com/Samourai-Wallet/addrindexrs"
repository = "https://github.com/romanz/electrs" repository = "https://github.com/Samourai-Wallet/addrindexrs"
keywords = ["bitcoin", "electrum", "server", "index", "database"] keywords = ["bitcoin", "server", "index", "database"]
documentation = "https://docs.rs/electrs/" documentation = "https://github.com/Samourai-Wallet/addrindexrs/blob/master/doc/usage.md"
readme = "README.md" readme = "README.md"
edition = "2018" edition = "2018"
build = "build.rs" build = "build.rs"
...@@ -37,7 +37,6 @@ log = "0.4" ...@@ -37,7 +37,6 @@ log = "0.4"
lru = "0.1" lru = "0.1"
num_cpus = "1.0" num_cpus = "1.0"
page_size = "0.4" page_size = "0.4"
prometheus = "0.5"
rocksdb = "0.12" rocksdb = "0.12"
rust-crypto = "0.2" rust-crypto = "0.2"
serde = "1.0" serde = "1.0"
......
...@@ -10,13 +10,4 @@ USER user ...@@ -10,13 +10,4 @@ USER user
WORKDIR /home/user WORKDIR /home/user
COPY ./ /home/user COPY ./ /home/user
RUN cargo build --release RUN cargo check
RUN cargo install --path .
# Electrum RPC
EXPOSE 50001
# Prometheus monitoring
EXPOSE 4224
STOPSIGNAL SIGINT
# Electrum Server in Rust # Bitcoin Address Indexer in Rust
[![CircleCI](https://circleci.com/gh/romanz/electrs/tree/master.svg?style=svg)](https://circleci.com/gh/romanz/electrs/tree/master) An efficient addresses indexer based on [Electrs](https://github.com/romanz/electrs) by [Roman Zeyde](https://github.com/romanz).
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
[![crates.io](http://meritbadge.herokuapp.com/electrs)](https://crates.io/crates/electrs)
[![gitter.im](https://badges.gitter.im/romanz/electrs.svg)](https://gitter.im/romanz/electrs)
An efficient re-implementation of Electrum Server, inspired by [ElectrumX](https://github.com/kyuupichan/electrumx), [Electrum Personal Server](https://github.com/chris-belcher/electrum-personal-server) and [bitcoincore-indexd](https://github.com/jonasschnelli/bitcoincore-indexd). The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries allowing to keep real-time track of the transaction history of Bitcoin addresses. Since it runs on the user's own machine, there is no need for the wallet to communicate with external servers, thus preserving the privacy of the user's addresses and balances.
The motivation behind this project is to enable a user to run his own Electrum server,
with required hardware resources not much beyond those of a [full node](https://en.bitcoin.it/wiki/Full_node#Why_should_you_use_a_full_node_wallet).
The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries for any given user wallet,
allowing the user to keep real-time track of his balances and his transaction history using the [Electrum wallet](https://electrum.org/).
Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers,
thus preserving the privacy of the user's addresses and balances.
## Features ## Features
* Supports Electrum protocol [v1.4](https://electrumx.readthedocs.io/en/latest/protocol.html) * Maintains an index over transaction inputs and outputs, allowing fast queries of the history of a Bitcoin address
* Maintains an index over transaction inputs and outputs, allowing fast balance queries
* Fast synchronization of the Bitcoin blockchain (~2 hours for ~187GB @ July 2018) on [modest hardware](https://gist.github.com/romanz/cd9324474de0c2f121198afe3d063548) * Fast synchronization of the Bitcoin blockchain (~2 hours for ~187GB @ July 2018) on [modest hardware](https://gist.github.com/romanz/cd9324474de0c2f121198afe3d063548)
* Low index storage overhead (~20%), relying on a local full node for transaction retrieval * Low index storage overhead (~20%), relying on a local full node for transaction retrieval
* Efficient mempool tracker (allowing better fee [estimation](https://github.com/spesmilo/electrum/blob/59c1d03f018026ac301c4e74facfc64da8ae4708/RELEASE-NOTES#L34-L46)) * Efficient mempool tracker
* Low CPU & memory usage (after initial indexing) * Low CPU & memory usage (after initial indexing)
* [`txindex`](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch03.asciidoc#txindex) is not required for the Bitcoin node
* Uses a single [RocksDB](https://github.com/spacejam/rust-rocksdb) database, for better consistency and crash recovery * Uses a single [RocksDB](https://github.com/spacejam/rust-rocksdb) database, for better consistency and crash recovery
## Usage ## Usage
......
# 0.8.0 (28 Oct 2019) # addrindexrs
## 0.1.0 (TBD)
# Prior releases (electrs)
## 0.8.0 (28 Oct 2019)
* Use `configure_me` instead of `clap` to support config files, environment variables and man pages (@Kixunil) * Use `configure_me` instead of `clap` to support config files, environment variables and man pages (@Kixunil)
* Don't accept `--cookie` via CLI arguments (@Kixunil) * Don't accept `--cookie` via CLI arguments (@Kixunil)
...@@ -7,12 +14,12 @@ ...@@ -7,12 +14,12 @@
* Bump rust-rocksdb to 0.12.3, using RockDB 6.1.2 * Bump rust-rocksdb to 0.12.3, using RockDB 6.1.2
* Bump bitcoin crate to 0.21 (@MichelKansou) * Bump bitcoin crate to 0.21 (@MichelKansou)
# 0.7.1 (27 July 2019) ## 0.7.1 (27 July 2019)
* Allow stopping bulk indexing via SIGINT/SIGTERM * Allow stopping bulk indexing via SIGINT/SIGTERM
* Cache list of transaction IDs for blocks (@dagurval) * Cache list of transaction IDs for blocks (@dagurval)
# 0.7.0 (13 June 2019) ## 0.7.0 (13 June 2019)
* Support Bitcoin Core 0.18 * Support Bitcoin Core 0.18
* Build with LTO * Build with LTO
...@@ -21,16 +28,16 @@ ...@@ -21,16 +28,16 @@
* Use atomics instead of `Mutex<u64>` (@Kixunil) * Use atomics instead of `Mutex<u64>` (@Kixunil)
* Better handling invalid blocks (@azuchi) * Better handling invalid blocks (@azuchi)
# 0.6.2 (17 May 2019) ## 0.6.2 (17 May 2019)
* Support Rust 1.32 (for Debian) * Support Rust 1.32 (for Debian)
# 0.6.1 (9 May 2019) ## 0.6.1 (9 May 2019)
* Fix crash during initial sync * Fix crash during initial sync
* Switch to `signal-hook` crate * Switch to `signal-hook` crate
# 0.6.0 (29 Apr 2019) ## 0.6.0 (29 Apr 2019)
* Update to Rust 1.34 * Update to Rust 1.34
* Prefix Prometheus metrics with 'electrs_' * Prefix Prometheus metrics with 'electrs_'
...@@ -40,35 +47,35 @@ ...@@ -40,35 +47,35 @@
* Fix "chain-trimming" reorgs * Fix "chain-trimming" reorgs
* Serve by default on IPv4 localhost * Serve by default on IPv4 localhost
# 0.5.0 (3 Mar 2019) ## 0.5.0 (3 Mar 2019)
* Limit query results, to prevent RPC server to get stuck (see `--txid-limit` flag) * Limit query results, to prevent RPC server to get stuck (see `--txid-limit` flag)
* Update RocksDB crate to 0.11 * Update RocksDB crate to 0.11
* Update Bitcoin crate to 0.17 * Update Bitcoin crate to 0.17
# 0.4.3 (23 Dec 2018) ## 0.4.3 (23 Dec 2018)
* Support Rust 2018 edition (1.31) * Support Rust 2018 edition (1.31)
* Upgrade to Electrum protocol 1.4 (from 1.2) * Upgrade to Electrum protocol 1.4 (from 1.2)
* Let server banner be configurable via command-line flag * Let server banner be configurable via command-line flag
* Improve query.get_merkle_proof() performance * Improve query.get_merkle_proof() performance
# 0.4.2 (22 Nov 2018) ## 0.4.2 (22 Nov 2018)
* Update to rust-bitcoin 0.15.1 * Update to rust-bitcoin 0.15.1
* Use bounded LRU cache for transaction retrieval * Use bounded LRU cache for transaction retrieval
* Support 'server.ping' and partially 'blockchain.block.header' Electrum RPC * Support 'server.ping' and partially 'blockchain.block.header' Electrum RPC
# 0.4.1 (14 Oct 2018) ## 0.4.1 (14 Oct 2018)
* Don't run full compaction after initial import is over (when using JSONRPC) * Don't run full compaction after initial import is over (when using JSONRPC)
# 0.4.0 (22 Sep 2018) ## 0.4.0 (22 Sep 2018)
* Optimize for low-memory systems by using different RocksDB settings * Optimize for low-memory systems by using different RocksDB settings
* Rename `--skip_bulk_import` flag to `--jsonrpc-import` * Rename `--skip_bulk_import` flag to `--jsonrpc-import`
# 0.3.2 (14 Sep 2018) ## 0.3.2 (14 Sep 2018)
* Optimize block headers processing during startup * Optimize block headers processing during startup
* Handle TCP disconnections during long RPCs * Handle TCP disconnections during long RPCs
...@@ -76,13 +83,12 @@ ...@@ -76,13 +83,12 @@
* Update rust-bitcoin to 0.14 * Update rust-bitcoin to 0.14
* Optimize block headers processing during startup * Optimize block headers processing during startup
## 0.3.1 (20 Aug 2018)
# 0.3.1 (20 Aug 2018)
* Reconnect to bitcoind only on transient errors * Reconnect to bitcoind only on transient errors
* Poll mempool after transaction broadcasting * Poll mempool after transaction broadcasting
# 0.3.0 (14 Aug 2018) ## 0.3.0 (14 Aug 2018)
* Optimize for low-memory systems * Optimize for low-memory systems
* Improve compaction performance * Improve compaction performance
...@@ -95,14 +101,14 @@ ...@@ -95,14 +101,14 @@
* Add some Python tools (as API usage examples) * Add some Python tools (as API usage examples)
* Change default Prometheus monitoring ports * Change default Prometheus monitoring ports
# 0.2.0 (14 Jul 2018) ## 0.2.0 (14 Jul 2018)
* Allow specifying custom bitcoind data directory * Allow specifying custom bitcoind data directory
* Allow specifying JSONRPC cookie from commandline * Allow specifying JSONRPC cookie from commandline
* Improve initial bulk indexing performance * Improve initial bulk indexing performance
* Support 32-bit systems * Support 32-bit systems
# 0.1.0 (2 Jul 2018) ## 0.1.0 (2 Jul 2018)
* Announcement: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-July/016190.html * Announcement: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-July/016190.html
* Published to https://crates.io/electrs and https://docs.rs/electrs * Published to https://crates.io/electrs and https://docs.rs/electrs
# Electrum # Indexer
* Snapshot DB after successful indexing - and run queries on the latest snapshot * Snapshot DB after successful indexing - and run queries on the latest snapshot
* Update height to -1 for txns with any [unconfirmed input](https://electrumx.readthedocs.io/en/latest/protocol-basics.html#status)
# Rust # Rust
......
[general] [general]
env_prefix = "ELECTRS" env_prefix = "ADDRINDEXRS"
conf_file_param = "conf" conf_file_param = "conf"
conf_dir_param = "conf_dir" conf_dir_param = "conf_dir"
doc = """ doc = """
An efficient re-implementation of Electrum Server, inspired by ElectrumX, Electrum Personal Server and bitcoincore-indexd. An efficient addresses indexer based on Electrs.
"""
The motivation behind this project is to enable a user to run his own Electrum server, with required hardware resources not much beyond those of a full node. The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries for any given user wallet, allowing the user to keep real-time track of his balances and his transaction history using the Electrum wallet. Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers, thus preserving the privacy of the user's addresses and balances."""
[[switch]] [[switch]]
name = "verbose" name = "verbose"
...@@ -42,20 +41,15 @@ doc = "Select Bitcoin network type ('mainnet', 'testnet' or 'regtest')" ...@@ -42,20 +41,15 @@ doc = "Select Bitcoin network type ('mainnet', 'testnet' or 'regtest')"
default = "Default::default()" default = "Default::default()"
[[param]] [[param]]
name = "electrum_rpc_addr" name = "indexer_rpc_addr"
type = "crate::config::ResolvAddr" type = "crate::config::ResolvAddr"
doc = "Electrum server JSONRPC 'addr:port' to listen on (default: '127.0.0.1:50001' for mainnet, '127.0.0.1:60001' for testnet and '127.0.0.1:60401' for regtest)" doc = "Indexer JSONRPC 'addr:port' to listen on (default: '127.0.0.1:50001' for mainnet, '127.0.0.1:60001' for testnet and '127.0.0.1:60401' for regtest)"
[[param]] [[param]]
name = "daemon_rpc_addr" name = "daemon_rpc_addr"
type = "crate::config::ResolvAddr" type = "crate::config::ResolvAddr"
doc = "Bitcoin daemon JSONRPC 'addr:port' to connect (default: 127.0.0.1:8332 for mainnet, 127.0.0.1:18332 for testnet and 127.0.0.1:18443 for regtest)" doc = "Bitcoin daemon JSONRPC 'addr:port' to connect (default: 127.0.0.1:8332 for mainnet, 127.0.0.1:18332 for testnet and 127.0.0.1:18443 for regtest)"
[[param]]
name = "monitoring_addr"
type = "crate::config::ResolvAddr"
doc = "Prometheus monitoring 'addr:port' to listen on (default: 127.0.0.1:4224 for mainnet, 127.0.0.1:14224 for testnet and 127.0.0.1:24224 for regtest)"
[[switch]] [[switch]]
name = "jsonrpc_import" name = "jsonrpc_import"
doc = "Use JSONRPC instead of directly importing blk*.dat files. Useful for remote full node or low memory system" doc = "Use JSONRPC instead of directly importing blk*.dat files. Useful for remote full node or low memory system"
...@@ -89,9 +83,3 @@ name = "txid_limit" ...@@ -89,9 +83,3 @@ name = "txid_limit"
type = "usize" type = "usize"
doc = "Number of transactions to lookup before returning an error, to prevent 'too popular' addresses from causing the RPC server to get stuck (0 - disable the limit)" doc = "Number of transactions to lookup before returning an error, to prevent 'too popular' addresses from causing the RPC server to get stuck (0 - disable the limit)"
default = "100" default = "100"
[[param]]
name = "server_banner"
type = "String"
doc = "The banner to be shown in the Electrum console"
default = "concat!(\"Welcome to electrs \", env!(\"CARGO_PKG_VERSION\"), \" (Electrum Rust Server)!\").to_owned()"
#!/usr/bin/env python3
import hashlib
import sys
import argparse
from pycoin.coins.bitcoin.networks import BitcoinTestnet, BitcoinMainnet
import client
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--testnet', action='store_true')
parser.add_argument('address', nargs='+')
args = parser.parse_args()
if args.testnet:
Network = BitcoinTestnet
port = 60001
else:
Network = BitcoinMainnet
port = 50001
conn = client.Client(('localhost', port))
for addr in args.address:
script = Network.ui.script_for_address(addr)
script_hash = hashlib.sha256(script).digest()[::-1].hex()
reply = conn.call('blockchain.scripthash.get_balance', script_hash)
result = reply['result']
print('{} has {} satoshis'.format(addr, result))
if __name__ == '__main__':
main()
import json
import socket
class Client:
def __init__(self, addr):
self.s = socket.create_connection(addr)
self.f = self.s.makefile('r')
self.id = 0
def call(self, method, *args):
req = {
'id': self.id,
'method': method,
'params': list(args),
}
msg = json.dumps(req) + '\n'
self.s.sendall(msg.encode('ascii'))
return json.loads(self.f.readline())
import binascii
import json
import os
import socket
class Daemon:
def __init__(self, port, cookie_dir):
self.sock = socket.create_connection(('localhost', port))
self.fd = self.sock.makefile()
path = os.path.join(os.path.expanduser(cookie_dir), '.cookie')
cookie = binascii.b2a_base64(open(path, 'rb').read())
self.cookie = cookie.decode('ascii').strip()
self.index = 0
def request(self, method, params_list):
obj = [{"method": method, "params": params, "id": self.index}
for params in params_list]
request = json.dumps(obj)
msg = ('POST / HTTP/1.1\n'
'Authorization: Basic {}\n'
'Content-Length: {}\n\n'
'{}'.format(self.cookie, len(request), request))
self.sock.sendall(msg.encode('ascii'))
status = self.fd.readline().strip()
while True:
if self.fd.readline().strip():
continue # skip headers
else:
break # next line will contain the response
data = self.fd.readline().strip()
replies = json.loads(data)
for reply in replies:
assert reply['error'] is None, reply
assert reply['id'] == self.index
self.index += 1
return [d['result'] for d in replies]
[Unit] [Unit]
Description=Electrum Rust Server Description=addrindexrs rust server
[Service] [Service]
Type=simple Type=simple
ExecStart=/path/to/electrs/target/release/electrs -vvvv --db-dir /path/to/electrs/db/ ExecStart=/path/to/addrindexrs/target/release/addrindexrs -vvvv --db-dir /path/to/addrindexrs/db/
Restart=on-failure Restart=on-failure
RestartSec=60 RestartSec=60
Environment="RUST_BACKTRACE=1" Environment="RUST_BACKTRACE=1"
......
#!/usr/bin/env python3
import argparse
from daemon import Daemon
import numpy as np
import matplotlib.pyplot as plt
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--testnet', action='store_true')
args = parser.parse_args()
if args.testnet:
d = Daemon(port=18332, cookie_dir='~/.bitcoin/testnet3')
else:
d = Daemon(port=8332, cookie_dir='~/.bitcoin')
txids, = d.request('getrawmempool', [[False]])
txids = list(map(lambda a: [a], txids))
entries = d.request('getmempoolentry', txids)
entries = [{'fee': e['fee']*1e8, 'vsize': e['vsize']} for e in entries]
for e in entries:
e['rate'] = e['fee'] / e['vsize'] # sat/vbyte
entries.sort(key=lambda e: e['rate'], reverse=True)
vsize = np.array([e['vsize'] for e in entries]).cumsum()
rate = np.array([e['rate'] for e in entries])
plt.semilogy(vsize / 1e6, rate, '-')
plt.xlabel('Mempool size (MB)')
plt.ylabel('Fee rate (sat/vbyte)')
plt.title('{} transactions'.format(len(entries)))
plt.grid()
plt.show()
if __name__ == '__main__':
main()
...@@ -3,7 +3,7 @@ set -eu ...@@ -3,7 +3,7 @@ set -eu
trap 'kill $(jobs -p)' EXIT trap 'kill $(jobs -p)' EXIT
DELAY=5 DELAY=5
LOG=/tmp/electrs.log LOG=/tmp/addrindexrs.log
CARGO="cargo +stable" CARGO="cargo +stable"
tail -v -n0 -F "$LOG" & tail -v -n0 -F "$LOG" &
......
#!/usr/bin/env python3
import argparse
import daemon
def main():
parser = argparse.ArgumentParser()
parser.add_argument('txid')
args = parser.parse_args()
d = daemon.Daemon(port=8332, cookie_dir='~/.bitcoin')
txid = args.txid
txn, = d.request('getrawtransaction', [[txid, True]])
vin = txn['vin']
fee = 0.0
for txi in txn['vin']:
prev_txid = txi['txid']
prev_tx, = d.request('getrawtransaction', [[prev_txid, True]])
index = txi['vout']
prev_txo = prev_tx['vout'][index]
print(f"{prev_txid}:{index:<5} {prev_txo['value']:+20.8f}")
fee += prev_txo['value']
for i, txo in enumerate(txn['vout']):
print(f"{txid}:{i:<5} {-txo['value']:+20.8f}")
fee -= txo['value']
print(f"Fee = {1e6 * fee:.2f} uBTC = {1e8 * fee / txn['vsize']:.2f} sat/vB")
if __name__ == '__main__':
main()
#!/usr/bin/env python3
import hashlib
import sys
from logbook import Logger, StreamHandler