Category Archives: CrytoCurrency

What to expect when running a Bitcoin Core Full Node

This post is not about how to install Bitcoin Core and run it as a full node.

Its about what you can expect when you install Bitcoin Core and run it as a full node.

About full nodes

Firstly, lets clarify the role of a full node. A node is any system that connects to the Bitcoin network. A full node is any system that connects to the Bitcoin network and retains a full and up to date copy of the blockchain.

A full node is not a miner. Nodes simply relay information. By contrast, a miner specifically listens for transactions and tries to create new blocks. Miners do not typically hold a copy of the blockchain.

A full node fulfils one of two roles.

If you simply run a full node, and do not use it as a wallet, all your node is doing is adding to the capacity of the Bitcoin Network to relay information between nodes. This is of some value, but not hugely significant.

If you run a full node and use it as a wallet (either directly or by linking a client wallet to your node), your full node adds to the economic strength of the Bitcoin Network. It does this by enforcing the consensus rules for transactions. The more nodes that enforce the consensus rules, the more difficult it is for malicious nodes to break that consensus.

It is also worth pointing out that running your Bitcoin transactions through your own node is the purest form of transacting in Bitcoin. You have your own copy of the blockchain and can verify transactions for which you are a beneficiary without have to rely on someone else’s copy of the blockchain. It is the most accurate interpretation of the common Bitcoin axiom of “being your own bank”.

Hardware

If you’re an individual and not employed by a software firm involved in Bitcoin, or some other agency tasked with promoting Bitcoin, chances are you’re going to run Bitcoin Core on some old system that you might otherwise have recycled/dumped.

Generally, this is OK. Where your system is going to need the most poke is in downloading the blockchain and verifying each block as it comes in. Once you have downloaded the entire blockchain, each new block is created every 10 minutes, so your system will have a 10 minute break between processing calls. Prior to this, when you’re downloading blocks one after the next in order to complete the blockchain, your system will exhaust its RAM and disk IO. Its quite normal for your system to be become momentarily unresponsive in this phase.

For reference, I downloaded the blockchain on an 8 year old mini-PC with 4GB of RAM and a 300GB disk. You’re going to need 180GB of disk at the time of writing to accommodate the current block chain.

Network

Something similar applies in respect of the network. The current blockchain is ~180GB, so you’re going to have to download this. There is no hurry with this. You can stop and start the Bitcoin daemon as often as you want. It will just pick up where it left off when you restart. I set up a cron schedule on mine to start the daemon at 23:00 and stop it again at 08:00, so that the node wasn’t interfering with my day to day download requirements. It took me 5-6 week to get the entire blockchain.

At the beginning blocks will will rack up really quickly, as the first blocks weren’t full and considerably smaller than the 1mb limit. As you get into the last 50k blocks (out of 500k at time of writing), where all blocks are full, things slow down significantly.

Once you have the entire chain, the load on the network eases, as you’re only picking up 1 new 1mb block every 10 minutes. There is also a bit of chatter re. notifications but nothing substantial.

One point to note:

If the Bitcoin daemon isn’t shut down cleanly, the next time it starts, it will re-verify all the blocks it downloaded during its last run. During this time, it won’t download any new blocks, and the RPC service won’t be available to process calls. If the previous run was particularly long, this process will also take a long time. You can check the log to see that this is happening. All you can do is let it run. If the daemon gets improperly killed again, the whole process will start again when the Bitcoin daemon is restarted. You should really, really try to avoid letting the daemon stop unexpectedly. Never kill the daemon.

Checking status

How do you know when you have downloaded the full blockchain?

One way you’ll know is when the Bitcoin daemon is running, but you disk isn’t thrashing and your system is generally responsive. That generally means you have all the blocks and the daemon is just sitting there waiting to pick up the next one that becomes available.

You can obviously verify this with an RPC call too:

root@ubuntu:~# bitcoin-cli getblockchaininfo | grep blocks -A 1
 "blocks": 508695,
 "headers": 508695,

This tell you that the service can see headers on the blockchain for 508,695, and also that there are 508,695 blocks on your system.

If you stop your system for  a few hours or days, and run this command again when your restart it, the number of blocks will be lower than the numbers of headers, and your system will start thrashing again as it catches up. The longer the gap, the longer the catch up period. When your system is catching up, it has no value to the Bitcoin network, so try and organise your system so that it is always on with cron controlling whether or not the Bitcoin daemon is running.

In my next post, I will explain how to use your full node with your Bitcoin wallet.

 

How Bitcoin mining pools work

I’ve written this to clarify my own understanding. Treat with caution.

Bitcoin mining pools exist because the computational power required to mine Bitcoins on a regular basis is so vast that it is beyond the financial and technical means of most people. Rather than investing a huge amount of money in mining equipment that will (hopefully) give you a return over a period of decades, a mining pool allows the individual to accumulate smaller amounts of Bitcoin more frequently.

The principle is quite straightforward: lots of people come together to combine their individual computing power into a single logical computing unit and share any rewards (Bitcoins) proportionally based on the amount of computing effort they contributed.

Where the complexity arises is in the regulation of the network, for instance:

How do you know how much effort was contributed by each member?

How do you prevent members jumping in and out of pools, particularly pools that haven’t mined a Bitcoin in while, and where it it likely they will mine a Bitcoin in the near future?

How do you prove that individual members are actually working?

How do you prevent more powerful members from hogging the network bandwidth of the master miner, preventing less powerful members from contributing?

How do you ensure that the master miner is getting enough throughput to have a reasonable chance of mining a Bitcoin?

There are just a sample of the problems that exist in relation to efficient and fair Bitcoin mining in pools. The following is a general explanation of how these problems are dealt with.

Let’s clarify terminology first (I’m assuming basic knowledge of how Bitcoin works here).

Hash: the end product, a binary number, of an encryption operation performed by a miner. Each new Hash is created by the miner adding a sequential string (a nonce) to the source (salt) of the encryption operation. Modern computers can generate hundreds of thousands of Hashes per second.

Target: In Bitcoin, each new Block has a Target. This is a binary number. To succeed in creating a new Block, a miner has to compute a Hash that is lower than that number. The Bitcoin protocol adjusts that number depending on the amount of activity on the network. If there is a lot of activity, the number becomes smaller, if there is less activity, the number becomes larger. The objective is to regulate the creation of new Blocks (ie Bitcoins) to 1 every 10 minutes or so.

Difficulty: This is a measure of how difficult it is for a miner to derive a Hash that is less than the current Target (ie mine Bitcoins). It is extrapolated from the amount of time it took to generate the last 2,016 blocks. At a rate of 1 block every 10 minutes, it should take 2 weeks to generate 2,016 blocks. If it has actually taken less than this, the difficulty will increase (ie the Target will be a smaller number). If it has taken more than this, the difficulty will decrease (ie the Target will be a larger number). The unit of measurement of difficulty is Hashes ie the number of Hashes on the entire network that were generated to created 2,016 Blocks.

Master Node: a full Bitcoin node that operates on the Bitcoin P2P network and which regulates a pool of members, who do not directly communicate on the Bitcoin P2P network, but who use the Master Node as a proxy.

Share: a Share is something that is particular to Mining pools. It does not form part of the wider Bitcoin protocol. It is the primary method used by the Master Node to regulate the activity of members of a pool. The next section deals with Shares.

When you start running a Bitcoin mining process, you will probably be aware of your Hash Rate. This is the number of Hashes your Bitcoin mining hardware is generating per second. These days, this is normally measured in Ghps, which means Millions (Giga) of Hashes Per Second. A typical high-end Graphics Card (GPU) on a modern PC can generate about 0.5 Ghps. A dedicated ASIC mining rig, which will cost over €1,000, might be able to generate 8,000 Ghps. A mining pool will typically measure its combined compute power in Thps, or Billions (Tera) of Hashes Per Second.

When you participate in a mining pool, and you see your hardware generating (lets say) 5,000 Ghps, this does not mean that you are submitting 5,000 Ghps to the Bitcoin network, via the Master Node. If that were the case, and all the pool members were doing the same, the Master Node that controls the pool would simply explode.

What it means is that your mining hardware can generate 5,000 Ghps locally on your computer.

This capacity isn’t used directly on the Bitcoin network. Instead, the Master Node that controls the pool acts as a proxy between the pool members and the main Bitcoin network. For this to work, the Master Node has to ensure both that the members are supplying enough Hashes for the Master Node to be able to compete on the main Bitcoin mining network, and that the allocation of any Bitcoin mined is divided proportionately according to the amount of compute effort supplied by the individual members.

To do this, the Master Node observes the Hash Rate of each of the members, and distributes computational challenges to them that all have a slightly lower Difficulty rating than the Difficultly rating of the current Block Target (lets call this the Proxy Target) . If such a Hash is found, the Master Node accepts this as a “Proof of Work Accepted”. If a Hash that is lower than the Proxy Target is not found within the allotted time, the member completes the work anyway before moving on to the next computational challenge distributed by the Master Node.

In this way, your mining process will log “Work Units Started”, which will always be a lower number than “Proof Of Works Accepted”. The smaller the gap between these numbers, the “luckier” you are. The greater the gap, the “unluckier” you are. In reality however, and over time, the gap should be consistent between across all members, as the master node will adjust the Difficultly of the computational challenges based on the Hash Rate of the member, which can change over time.

A Share is therefore the equivalent of a “Proof of Work Accepted”. The Master Node will keep a record of all “Proof of Works Accepted” from each member, and distribute Bitcoin mined based on the number of Shares each member has contributed.

Confused? Of course you are, so lets go through that again, from a different perspective.

If the Master Node were sending computational challenges to members that had a Difficulty rating that was equal to the Difficulty rating of the current Target in the Blockchain, the Master Node would just be sitting there idlly for days on end waiting for one of the members to come up with the necessary Hash to create the new Block. The Master Node would have no knowledge of what effort the other members contributed, and would have no option but to award the full reward to the successful member, even if that member only contributed 0.001% of computational effort involved in creating the Block.

Instead, the Master Node lowers the bar on the Difficulty rating (relative to the actual difficulty rating of the current Block) so that it receives lots of Hashes from the members. All but one of these Hashes will be lower than the current Blockchain target, but at least now the Master Node can confirm that its members are working, and at what rate they are working. It can then use that information to both regulate the traffic received from the members and proportionately divide any rewards.

Additionally, it can ensure that more powerful members, whose submissions are rate-limited to allow submissions from less powerful members, are not discriminated against. These more powerful nodes are given challenges with higher Difficulty ratings, but any “Proof of Work Submitted”s (ie Shares) that they accumulate are weighted according to the Difficulty rating that was set, giving them a higher proportionate of any ultimate reward.

While all  of the above may sound complex, it is still in fact just a general introduction to mining pools, and based on the mining pool that I use, Bitminter.

Other pools use variations of this methodology, but all follow the general principle that the Master Node is a proxy to the main Bitcoin network, that members must prove to the Master Node that they are working and that rewards are allocated based on the amount of work done.

A brief note about payment methodologies is also warranted. Many pools will use either the PPS (Pay Per Share) or PPLNS (Pay Per Last N Shares) method to distribute rewards.

In the PPS model, you get a payment for each Share you contribute regardless of the success or failure of the mining pool. This makes for a regular income, but doesn’t allow you to benefit when the pool has a lucky streak.

In the PPLNS model, you get paid only when Bitcoins are mined, and only on the basis of the Shares you submitted to that effort. This makes for more irregular income, but allows you to benefit when the pool has a lucky streak.