Unverified Commit 31e7539e authored by kenshin samourai's avatar kenshin samourai Committed by GitHub
Browse files

Merge pull request #3 from kenshin-samourai/develop_dojo

init master for addrindexrs
parents 4d0f52b8 eba463c9
version: 2
jobs:
build:
docker:
- image: rust:1.34.0-slim
steps:
- checkout
- run:
name: Install Dependencies
command: |
apt-get update
apt-get install -y clang cmake libsnappy-dev
rustup component add rustfmt
rustup component add clippy
- run:
name: Format
command: cargo fmt --all -- --check
- run:
name: Check
command: cargo check --all
- run:
name: Clippy
command: cargo clippy --all
- run:
name: Build
command: cargo build --all
- run:
name: Test
command: cargo test --all
target/
.git/
_*/
......@@ -5,3 +5,4 @@ _*/
*.sublime*
*~
*.pyc
Dockerfile
language: rust
rust:
- stable
cache: cargo
before_script:
- rustup component add rustfmt-preview
- rustup component add clippy
script:
- cargo fmt --all -- --check
- cargo check --all
- cargo clippy --all
- cargo build --all
- cargo test --all
This diff is collapsed.
[package]
name = "electrs"
version = "0.8.0"
authors = ["Roman Zeyde <me@romanzey.de>"]
description = "An efficient re-implementation of Electrum Server in Rust"
name = "addrindexrs"
version = "0.1.0"
authors = ["Roman Zeyde <me@romanzey.de>", "kenshin-samourai <kenshin_samourai@tutanota.com>"]
description = "An efficient address indexer in Rust"
license = "MIT"
homepage = "https://github.com/romanz/electrs"
repository = "https://github.com/romanz/electrs"
keywords = ["bitcoin", "electrum", "server", "index", "database"]
documentation = "https://docs.rs/electrs/"
homepage = "https://github.com/Samourai-Wallet/addrindexrs"
repository = "https://github.com/Samourai-Wallet/addrindexrs"
keywords = ["bitcoin", "server", "index", "database"]
documentation = "https://github.com/Samourai-Wallet/addrindexrs/blob/master/doc/usage.md"
readme = "README.md"
edition = "2018"
build = "build.rs"
......@@ -37,7 +37,6 @@ log = "0.4"
lru = "0.1"
num_cpus = "1.0"
page_size = "0.4"
prometheus = "0.5"
rocksdb = "0.12"
rust-crypto = "0.2"
serde = "1.0"
......
......@@ -10,13 +10,4 @@ USER user
WORKDIR /home/user
COPY ./ /home/user
RUN cargo build --release
RUN cargo install --path .
# Electrum RPC
EXPOSE 50001
# Prometheus monitoring
EXPOSE 4224
STOPSIGNAL SIGINT
RUN cargo check
# Electrum Server in Rust
# Bitcoin Address Indexer in Rust
[![CircleCI](https://circleci.com/gh/romanz/electrs/tree/master.svg?style=svg)](https://circleci.com/gh/romanz/electrs/tree/master)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
[![crates.io](http://meritbadge.herokuapp.com/electrs)](https://crates.io/crates/electrs)
[![gitter.im](https://badges.gitter.im/romanz/electrs.svg)](https://gitter.im/romanz/electrs)
An efficient addresses indexer based on [Electrs](https://github.com/romanz/electrs) by [Roman Zeyde](https://github.com/romanz).
An efficient re-implementation of Electrum Server, inspired by [ElectrumX](https://github.com/kyuupichan/electrumx), [Electrum Personal Server](https://github.com/chris-belcher/electrum-personal-server) and [bitcoincore-indexd](https://github.com/jonasschnelli/bitcoincore-indexd).
The motivation behind this project is to enable a user to run his own Electrum server,
with required hardware resources not much beyond those of a [full node](https://en.bitcoin.it/wiki/Full_node#Why_should_you_use_a_full_node_wallet).
The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries for any given user wallet,
allowing the user to keep real-time track of his balances and his transaction history using the [Electrum wallet](https://electrum.org/).
Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers,
thus preserving the privacy of the user's addresses and balances.
The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries allowing to keep real-time track of the transaction history of Bitcoin addresses. Since it runs on the user's own machine, there is no need for the wallet to communicate with external servers, thus preserving the privacy of the user's addresses and balances.
## Features
* Supports Electrum protocol [v1.4](https://electrumx.readthedocs.io/en/latest/protocol.html)
* Maintains an index over transaction inputs and outputs, allowing fast balance queries
* Maintains an index over transaction inputs and outputs, allowing fast queries of the history of a Bitcoin address
* Fast synchronization of the Bitcoin blockchain (~2 hours for ~187GB @ July 2018) on [modest hardware](https://gist.github.com/romanz/cd9324474de0c2f121198afe3d063548)
* Low index storage overhead (~20%), relying on a local full node for transaction retrieval
* Efficient mempool tracker (allowing better fee [estimation](https://github.com/spesmilo/electrum/blob/59c1d03f018026ac301c4e74facfc64da8ae4708/RELEASE-NOTES#L34-L46))
* Efficient mempool tracker
* Low CPU & memory usage (after initial indexing)
* [`txindex`](https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch03.asciidoc#txindex) is not required for the Bitcoin node
* Uses a single [RocksDB](https://github.com/spacejam/rust-rocksdb) database, for better consistency and crash recovery
## Usage
......
# 0.8.0 (28 Oct 2019)
# addrindexrs
## 0.1.0 (TBD)
# Prior releases (electrs)
## 0.8.0 (28 Oct 2019)
* Use `configure_me` instead of `clap` to support config files, environment variables and man pages (@Kixunil)
* Don't accept `--cookie` via CLI arguments (@Kixunil)
......@@ -7,12 +14,12 @@
* Bump rust-rocksdb to 0.12.3, using RockDB 6.1.2
* Bump bitcoin crate to 0.21 (@MichelKansou)
# 0.7.1 (27 July 2019)
## 0.7.1 (27 July 2019)
* Allow stopping bulk indexing via SIGINT/SIGTERM
* Cache list of transaction IDs for blocks (@dagurval)
# 0.7.0 (13 June 2019)
## 0.7.0 (13 June 2019)
* Support Bitcoin Core 0.18
* Build with LTO
......@@ -21,16 +28,16 @@
* Use atomics instead of `Mutex<u64>` (@Kixunil)
* Better handling invalid blocks (@azuchi)
# 0.6.2 (17 May 2019)
## 0.6.2 (17 May 2019)
* Support Rust 1.32 (for Debian)
# 0.6.1 (9 May 2019)
## 0.6.1 (9 May 2019)
* Fix crash during initial sync
* Switch to `signal-hook` crate
# 0.6.0 (29 Apr 2019)
## 0.6.0 (29 Apr 2019)
* Update to Rust 1.34
* Prefix Prometheus metrics with 'electrs_'
......@@ -40,35 +47,35 @@
* Fix "chain-trimming" reorgs
* Serve by default on IPv4 localhost
# 0.5.0 (3 Mar 2019)
## 0.5.0 (3 Mar 2019)
* Limit query results, to prevent RPC server to get stuck (see `--txid-limit` flag)
* Update RocksDB crate to 0.11
* Update Bitcoin crate to 0.17
# 0.4.3 (23 Dec 2018)
## 0.4.3 (23 Dec 2018)
* Support Rust 2018 edition (1.31)
* Upgrade to Electrum protocol 1.4 (from 1.2)
* Let server banner be configurable via command-line flag
* Improve query.get_merkle_proof() performance
# 0.4.2 (22 Nov 2018)
## 0.4.2 (22 Nov 2018)
* Update to rust-bitcoin 0.15.1
* Use bounded LRU cache for transaction retrieval
* Support 'server.ping' and partially 'blockchain.block.header' Electrum RPC
# 0.4.1 (14 Oct 2018)
## 0.4.1 (14 Oct 2018)
* Don't run full compaction after initial import is over (when using JSONRPC)
# 0.4.0 (22 Sep 2018)
## 0.4.0 (22 Sep 2018)
* Optimize for low-memory systems by using different RocksDB settings
* Rename `--skip_bulk_import` flag to `--jsonrpc-import`
# 0.3.2 (14 Sep 2018)
## 0.3.2 (14 Sep 2018)
* Optimize block headers processing during startup
* Handle TCP disconnections during long RPCs
......@@ -76,13 +83,12 @@
* Update rust-bitcoin to 0.14
* Optimize block headers processing during startup
# 0.3.1 (20 Aug 2018)
## 0.3.1 (20 Aug 2018)
* Reconnect to bitcoind only on transient errors
* Poll mempool after transaction broadcasting
# 0.3.0 (14 Aug 2018)
## 0.3.0 (14 Aug 2018)
* Optimize for low-memory systems
* Improve compaction performance
......@@ -95,14 +101,14 @@
* Add some Python tools (as API usage examples)
* Change default Prometheus monitoring ports
# 0.2.0 (14 Jul 2018)
## 0.2.0 (14 Jul 2018)
* Allow specifying custom bitcoind data directory
* Allow specifying JSONRPC cookie from commandline
* Improve initial bulk indexing performance
* Support 32-bit systems
# 0.1.0 (2 Jul 2018)
## 0.1.0 (2 Jul 2018)
* Announcement: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-July/016190.html
* Published to https://crates.io/electrs and https://docs.rs/electrs
# Electrum
# Indexer
* Snapshot DB after successful indexing - and run queries on the latest snapshot
* Update height to -1 for txns with any [unconfirmed input](https://electrumx.readthedocs.io/en/latest/protocol-basics.html#status)
# Rust
......
[general]
env_prefix = "ELECTRS"
env_prefix = "ADDRINDEXRS"
conf_file_param = "conf"
conf_dir_param = "conf_dir"
doc = """
An efficient re-implementation of Electrum Server, inspired by ElectrumX, Electrum Personal Server and bitcoincore-indexd.
The motivation behind this project is to enable a user to run his own Electrum server, with required hardware resources not much beyond those of a full node. The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries for any given user wallet, allowing the user to keep real-time track of his balances and his transaction history using the Electrum wallet. Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers, thus preserving the privacy of the user's addresses and balances."""
An efficient addresses indexer based on Electrs.
"""
[[switch]]
name = "verbose"
......@@ -33,9 +32,6 @@ default = "crate::config::default_daemon_dir()"
name = "cookie"
type = "String"
doc = "JSONRPC authentication cookie ('USER:PASSWORD', default: read from ~/.bitcoin/.cookie)"
# Force the user to use config file in order to avoid password leaks
argument = false
env_var = false
[[param]]
name = "network"
......@@ -45,20 +41,15 @@ doc = "Select Bitcoin network type ('mainnet', 'testnet' or 'regtest')"
default = "Default::default()"
[[param]]
name = "electrum_rpc_addr"
name = "indexer_rpc_addr"
type = "crate::config::ResolvAddr"
doc = "Electrum server JSONRPC 'addr:port' to listen on (default: '127.0.0.1:50001' for mainnet, '127.0.0.1:60001' for testnet and '127.0.0.1:60401' for regtest)"
doc = "Indexer JSONRPC 'addr:port' to listen on (default: '127.0.0.1:50001' for mainnet, '127.0.0.1:60001' for testnet and '127.0.0.1:60401' for regtest)"
[[param]]
name = "daemon_rpc_addr"
type = "crate::config::ResolvAddr"
doc = "Bitcoin daemon JSONRPC 'addr:port' to connect (default: 127.0.0.1:8332 for mainnet, 127.0.0.1:18332 for testnet and 127.0.0.1:18443 for regtest)"
[[param]]
name = "monitoring_addr"
type = "crate::config::ResolvAddr"
doc = "Prometheus monitoring 'addr:port' to listen on (default: 127.0.0.1:4224 for mainnet, 127.0.0.1:14224 for testnet and 127.0.0.1:24224 for regtest)"
[[switch]]
name = "jsonrpc_import"
doc = "Use JSONRPC instead of directly importing blk*.dat files. Useful for remote full node or low memory system"
......@@ -92,9 +83,3 @@ name = "txid_limit"
type = "usize"
doc = "Number of transactions to lookup before returning an error, to prevent 'too popular' addresses from causing the RPC server to get stuck (0 - disable the limit)"
default = "100"
[[param]]
name = "server_banner"
type = "String"
doc = "The banner to be shown in the Electrum console"
default = "concat!(\"Welcome to electrs \", env!(\"CARGO_PKG_VERSION\"), \" (Electrum Rust Server)!\").to_owned()"
#!/usr/bin/env python3
import hashlib
import sys
import argparse
from pycoin.coins.bitcoin.networks import BitcoinTestnet, BitcoinMainnet
import client
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--testnet', action='store_true')
parser.add_argument('address', nargs='+')
args = parser.parse_args()
if args.testnet:
Network = BitcoinTestnet
port = 60001
else:
Network = BitcoinMainnet
port = 50001
conn = client.Client(('localhost', port))
for addr in args.address:
script = Network.ui.script_for_address(addr)
script_hash = hashlib.sha256(script).digest()[::-1].hex()
reply = conn.call('blockchain.scripthash.get_balance', script_hash)
result = reply['result']
print('{} has {} satoshis'.format(addr, result))
if __name__ == '__main__':
main()
import json
import socket
class Client:
def __init__(self, addr):
self.s = socket.create_connection(addr)
self.f = self.s.makefile('r')
self.id = 0
def call(self, method, *args):
req = {
'id': self.id,
'method': method,
'params': list(args),
}
msg = json.dumps(req) + '\n'
self.s.sendall(msg.encode('ascii'))
return json.loads(self.f.readline())
import binascii
import json
import os
import socket
class Daemon:
def __init__(self, port, cookie_dir):
self.sock = socket.create_connection(('localhost', port))
self.fd = self.sock.makefile()
path = os.path.join(os.path.expanduser(cookie_dir), '.cookie')
cookie = binascii.b2a_base64(open(path, 'rb').read())
self.cookie = cookie.decode('ascii').strip()
self.index = 0
def request(self, method, params_list):
obj = [{"method": method, "params": params, "id": self.index}
for params in params_list]
request = json.dumps(obj)
msg = ('POST / HTTP/1.1\n'
'Authorization: Basic {}\n'
'Content-Length: {}\n\n'
'{}'.format(self.cookie, len(request), request))
self.sock.sendall(msg.encode('ascii'))
status = self.fd.readline().strip()
while True:
if self.fd.readline().strip():
continue # skip headers
else:
break # next line will contain the response
data = self.fd.readline().strip()
replies = json.loads(data)
for reply in replies:
assert reply['error'] is None, reply
assert reply['id'] == self.index
self.index += 1
return [d['result'] for d in replies]
[Unit]
Description=Electrum Rust Server
Description=addrindexrs rust server
[Service]
Type=simple
ExecStart=/path/to/electrs/target/release/electrs -vvvv --db-dir /path/to/electrs/db/
ExecStart=/path/to/addrindexrs/target/release/addrindexrs -vvvv --db-dir /path/to/addrindexrs/db/
Restart=on-failure
RestartSec=60
Environment="RUST_BACKTRACE=1"
......
#!/usr/bin/env python3
import argparse
from daemon import Daemon
import numpy as np
import matplotlib.pyplot as plt
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--testnet', action='store_true')
args = parser.parse_args()
if args.testnet:
d = Daemon(port=18332, cookie_dir='~/.bitcoin/testnet3')
else:
d = Daemon(port=8332, cookie_dir='~/.bitcoin')
txids, = d.request('getrawmempool', [[False]])
txids = list(map(lambda a: [a], txids))
entries = d.request('getmempoolentry', txids)
entries = [{'fee': e['fee']*1e8, 'vsize': e['vsize']} for e in entries]
for e in entries:
e['rate'] = e['fee'] / e['vsize'] # sat/vbyte
entries.sort(key=lambda e: e['rate'], reverse=True)
vsize = np.array([e['vsize'] for e in entries]).cumsum()
rate = np.array([e['rate'] for e in entries])
plt.semilogy(vsize / 1e6, rate, '-')
plt.xlabel('Mempool size (MB)')
plt.ylabel('Fee rate (sat/vbyte)')
plt.title('{} transactions'.format(len(entries)))
plt.grid()
plt.show()
if __name__ == '__main__':
main()
......@@ -3,7 +3,7 @@ set -eu
trap 'kill $(jobs -p)' EXIT
DELAY=5
LOG=/tmp/electrs.log
LOG=/tmp/addrindexrs.log
CARGO="cargo +stable"
tail -v -n0 -F "$LOG" &
......
#!/usr/bin/env python3
import argparse
import daemon
def main():
parser = argparse.ArgumentParser()
parser.add_argument('txid')
args = parser.parse_args()
d = daemon.Daemon(port=8332, cookie_dir='~/.bitcoin')
txid = args.txid
txn, = d.request('getrawtransaction', [[txid, True]])
vin = txn['vin']
fee = 0.0
for txi in txn['vin']:
prev_txid = txi['txid']
prev_tx, = d.request('getrawtransaction', [[prev_txid, True]])
index = txi['vout']
prev_txo = prev_tx['vout'][index]
print(f"{prev_txid}:{index:<5} {prev_txo['value']:+20.8f}")
fee += prev_txo['value']
for i, txo in enumerate(txn['vout']):
print(f"{txid}:{i:<5} {-txo['value']:+20.8f}")
fee -= txo['value']
print(f"Fee = {1e6 * fee:.2f} uBTC = {1e8 * fee / txn['vsize']:.2f} sat/vB")
if __name__ == '__main__':
main()
#!/usr/bin/env python3
import hashlib
import sys
from logbook import Logger, StreamHandler
from pycoin.coins.bitcoin.networks import BitcoinMainnet
import pycoin.ui.key_from_text
import pycoin.key
import client
script_for_address = BitcoinMainnet.ui.script_for_address
log = Logger(__name__)
def main():
conn = client.Client(('localhost', 50001))
xpub, = sys.argv[1:]
total = 0
k = pycoin.ui.key_from_text.key_from_text(xpub)
for change in (0, 1):
empty = 0
for n in range(100):
address = k.subkey(change).subkey(n).address()
script = script_for_address(address)
script_hash = hashlib.sha256(script).digest()[::-1].hex()
log.debug('{}', conn.call('blockchain.scripthash.get_history',
script_hash))
reply = conn.call('blockchain.scripthash.get_balance', script_hash)
result = reply['result']
confirmed = result['confirmed'] / 1e8
total += confirmed
if confirmed:
log.info('{}/{} => {} has {:11.8f} BTC',
change, n, address, confirmed)
empty = 0
else:
empty += 1
if empty >= 10:
break
log.info('total balance: {} BTC', total)
if __name__ == '__main__':
with StreamHandler(sys.stderr, level='INFO').applicationbound():
main()
......@@ -6,9 +6,9 @@ The index is stored at a single RocksDB database using the following schema:
Allows efficiently finding all funding transactions for a specific address:
| Code | Script Hash Prefix | Funding TxID Prefix | |
| ------ | -------------------- | --------------------- | - |
| `b'O'` | `SHA256(script)[:8]` | `txid[:8]` | |
| Code | Script Hash Prefix | Funding TxID Prefix | Funding Output Index | |
| ------ | -------------------- | --------------------- | --------------------- | - |
| `b'O'` | `SHA256(script)[:8]` | `txid[:8]` | `uint16` | |
## Transaction inputs' index
......@@ -23,9 +23,15 @@ Allows efficiently finding spending transaction of a specific output:
In order to save storage space, we store the full transaction IDs once, and use their 8-byte prefixes for the indexes above.
| Code | Transaction ID | | Confirmed height |
| ------ | ----------------- | - | ------------------ |
| `b'T'` | `txid` (32 bytes) | | `uint32` |
| Code | Transaction ID | |
| ------ | ----------------- | - |
| `b'T'` | `txid` (32 bytes) | |
## Blocks
Stores the hashes and headers of blocks.
Note that this mapping allows us to use `getrawtransaction` RPC to retrieve actual transaction data from without `-txindex` enabled
(by explicitly specifying the [blockhash](https://github.com/bitcoin/bitcoin/commit/497d0e014cc79d46531d570e74e4aeae72db602d)).
| Code | Block hash | | Block header |
| ------ | ----------------- | - | --------------------- |
| `b'B'` | `hash` (32 bytes) | | 80 bytes |
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment