What is Sharding?

Introduction

In a conventional blockchain, transactions must be confirmed by each node in the network while a consensus needs to be reached among the nodes before the transactions are packed to generate a new block. Security is fully guaranteed as each node records the complete ledger history, making it difficult for hackers to surreptitiously tamper with the data without being detected.
Security and scalability are incompatible
However, high security comes at a price. The frequency of nodes communicating and information exchanging occupies network bandwidth and slows down transaction processing. In order to prevent the failure of a single node from halting the whole network, more nodes are needed to enhance decentralization and disperse risk. These different trade-offs are also known as the “impossible triangle”, which means that the scalability, decentralization and security of the blockchain network cannot be achieved at the same time.

Source: Vitalik Buterin’s blog “Why sharding is great: demystifying the technical properties”

Sharding enables efficient scaling while achieving security and decentralization
Sharding is a solution that increases the scalability of a blockchain network without giving up security and decentralization. Sharding splits a single blockchain into multiple smaller blockchains, each of which manages its own records separately, and date exchange is only performed when necessary. Therefore, nodes in different sharding chains can verify only the transactions belonging to their own chains synchronously, instead of data from the entire blockchain.
As long as the number of nodes on each sharding chain is large enough, the security and decentralization will be confirmed. The application of sharding can increase the throughput of blockchains to meet the increasing user demand. ETH 2.0 will also enhance its scalability through sharding.

What is sharding?

Sharding is an optimization technique to improve database access performance by cutting the original database into multiple shards based on demand conditions, which can spread load and user traffic to provide faster service. Considering that a blockchain is essentially a large decentralized ledger, it is possible to improve scalability through sharding.
For example, there are a large number of products with different prices on shopping sites, consumers come with a variety of needs while some prefer discounted and cheap products and others like expensive and premium ones. To deal with a large order like this, a simple triage can be done using price ranges to speed up checkers’ processing.

Source: DigitalOcean
Sharding triages data and processes it according to different needs
A checker (node) has to confirm all the order information without splitting the order data (sharding), which takes a lot of time to perform repetitive tasks. By contrast, dividing the order data into several shards allows different checkers to process orders for low-priced and high-priced ones separately.
Since each type of order is a small part of the complete order, and checkers in charge for different priced items can work simultaneously without affecting each other, so sharding allows parallel operations; each checker only needs to check part of the order, so sharding reduces the workload of nodes and speed up verification.
A sharded database is like a puzzle, and each piece of the puzzle represents a shard. An unsharded blockchain has to redraw the entire puzzle every time a new block is generated (state replication) and then modify a small part of the puzzle (state update). A sharded blockchain only needs to find the specific piece requiring modification (shard) and replace it with a new one when adding a new transaction record.

Why is Sharding Necessary?

On-chain applications are more diverse, and demand grows exponentially
Once there is a significant increase in user size, no matter what kind the system is, it inevitably needs to scale to accommodate the increasing traffic. For example, if an online game is so popular that the number of players increases from 100,000 to a million or even ten million per day, it is necessary to set up a new server to divert traffic and avoid lag. A similar situation can be seen in real life. During holidays, tourists to places of interest always suffer from terrible traffic jams. What might normally take only two hours to reach a destination now takes much longer due to the surge in traffic. Alternative routines are the common solution to effectively ease the traffic congestion.
Scaling encounters bottlenecks, looking for alternatives
Blockchain has also encountered the bottleneck of scalability during its development. In a P2P blockchain with only 10 nodes, all data exchange can be completed after 109 times of inter-node communication; when the number of nodes reaches 100, it means 10099 times of inter-node communication if there is no algorithm optimization. In general, a P2P blockchain network with N nodes, where each node exchanges data with other N-1 nodes, it is conceivable that the amount of time and computation spent on each transaction will grow exponentially following the scaling of blockchain networks.

Through being the mainstream, Bitcoin and Ethereum are still applying outdated mechanism
Bitcoin, currently the largest cryptocurrency by market capitalization, and pre-upgraded Ethereum are both low-speed, with Bitcoin processing about 7 transactions per second (TPS) and Ethereum a bit more at 15 TPS. This has long been insufficient to meet the needs of a large number of users, and is dwarfed by the centralized system such as leading credit card company VISA, which processes up to 24,000 transactions per second. In addition to network congestion, the slow speed also increases costs, and users need to pay higher fees to prioritize their transactions, making the user experience even worse.

Source: Etherscan

There is no need for each node to own the complete transaction history in a sharded blockchain as a node only needs to retain information related to the shard it belongs to, the cost of setting up nodes and the threshold for participation are lowered by sharding. Blockchain networks are possible to become more decentralized with the help of sharding. The specifications of hardware equipment required to run nodes will become higher and higher without sharding. In the end, only participants with favorable economic conditions will survive. Sharding enables ordinary computers, laptops, and even smartphones to become nodes, which will lead to the mass adoption of the blockchains and dAPPs.

How Sharding Works in ETH 2.0?

The consensus mechanism of Ethereum will be converted from Proof of Work (PoW) to Proof of Stake (PoS) in ETH 2.0 and the original Ethereum mainnet will be merged into the Beacon Chain. As a result, massive computing power is no longer needed for Ethereum nodes to generate new blocks. Instead, they stake their ETH in smart contracts to become validators to get the right to obtain transaction fees when submitting new blocks.
In ETH 2.0, 64 shards will be created to improve scalability, one of which is called the Beacon Chain. It plays a core role in Ethereum sharding upgrades and is responsible for coordinating and information sharing between different shards. All the ETH 2.0 nodes will also stake ETHs on the shard chains and process transactions on different shards according to instructions, and blocks on the shard chains will only be valid with the approval of the Beacon Chain.

Souce: Hsiao-wei Wang
Sharding validates pending transactions by triage
When ETH 2.0 adopts sharding, the random sampling algorithm will assign random numbers of nodes to each shard to verify transactions and determine the order and validity of the transactions on the shard chain by voting, then the information of the new-generated shard blocks will be added on the Beacon Chain. A new block must be approved by more than two-thirds of the nodes on the shard chain.
As shown in the figure below, each node is assigned a number (1~100) in a sequence of validator node sets; the random sampling algorithm will break up the assignment order of the nodes, resulting in a new node set with no fixed order; the new-generated first 1~10 nodes can be designated as the first committee to process transactions on the first shard chain while the 11th~20th being the second committee to process the second shard chain, and so on.

Source: Vitalik Buterin’s blog “Why sharding is great: demystifying the technical properties”
The validator node is not fixedly assigned to process transactions on a certain shard chain as the random sampling algorithm will re-disrupt the order of nodes for a period of time. Therefore, the committee members responsible for each shard chain are not fixed, which avoids the risk of centralization caused by a specific node verifying a shard chain for a long time, and increases the difficulty of attacking.
Since sharding will greatly increase the throughput of ETH 2.0, it becomes infeasible for all nodes to update the complete transaction data on different shard chains synchronously, the Collation Header, which is similar to the block header in the PoW , becomes the medium for information sharing. As the name implies, a collation header contains metadata about the information inside the collation such as:

1. The single shard that the collation belongs to
2. The root hash of the parent collation
3. The Merkle root of all transactions in a collation
4. The pre-state root and post-state root
5. Signatures of notaries

Source: Hackernoon
The collation header will provide the node with sufficient information and will only download the complete transaction record according to the index when necessary.

Pros and Cons of Sharding

An alternative that effectively diverting usage demand and significantly improving scalability
Imagine that a million cars are forced to drive on the same road to move from city A to city B. The journey is bound to be slow and that is the case for conventional blockchains as all on-chain nodes are required to verify every transaction. Sharding slices on-chain data into pieces so that each node only needs to process a portion of the transaction. When a blockchain adds multiple highways to disperse the traffic, users can rightfully complete transactions faster.

Source: Genesis Block

Reduce network congestion and lower the costs
In pre-upgrade Ethereum, miner nodes pack transactions based on how much users are willing to pay, with higher fees being processed faster and lower fees being processed slower, resulting in a pathological fee bidding where users occasionally pay more than $50 for a single transaction. Sharding allows the blockchain to operate more smoothly and reduces the necessity for fee bidding, and users can turn to another one if congestion occurs on one shard.

Easier to run nodes, enhancing decentralization and security
Since it is not necessary to access the complete data of the whole blockchain as each node only needs to process information on one shard at the same time, the hardware required to set up a node becomes lower. Users are able to use cheaper devices as nodes to join in verification so as to earn revenue. The decreasing threshold benefits the decentralization and popularization of blockchain networks, and the vision of ETH 2.0 is to allow individuals to run Ethereum apps from their smartphones. A side benefit of decentralization is the enhancement of network security, the more nodes and the more dispersed they are, the more difficult it is to attack the blockchain.

While sharding highlights a list of advantages, it also introduces a series of new problems:

The possibility of the 1% Attack
The blockchain is vulnerable to 51% attacks, which means an attacker is able to arbitrarily tamper with transaction data or even control the whole blockchain network when he controls 51% of the nodes’ computing power. In larger blockchain networks such as Bitcoin and pre-upgrade Ethereum, the cost of a 51% attack is too high, making it infeasible to implement.
However, with the adoption of sharding, the number of nodes processing each shard is drastically reduced and it becomes much easier for a malicious attacker to compromise as long as he attacks one of the shards and forge the transactions successfully rather than attacking the whole network. To be more specific, for a blockchain network containing 1,000 nodes with the same computing power, a hacker needs to control more than 500 nodes to attack the whole network. But when these 1,000 nodes are distributed to 100 shards, a hacker only needs to control more than 5 nodes in one of the shards to tamper with the data.

Source: Genesis Block

Risk of smart contract security increases
The adoption of sharding requires rewriting the data structure and underlying code logic of the blockchain network ledger, and the further complication will make it more difficult to update and maintain the network, resulting in security vulnerabilities and risks of smart contracts while unintended errors may also occur during execution.

Collusion among committee members
Although sharding adopts a random sampling algorithm to avoid identical nodes being assigned to a fixed shard chain, there is still a possibility that duplicate combinations of identical nodes can co-validate transactions after a long enough period of time, and committee members verifying the shard chain are also possible to collude with each other and submit malicious transaction to the chain.
Load imbalance
The prerequisite for improving network performance through sharding is that “sharding” successfully brings “diversion”. When a blockchain is optimized with 100 shard chains, however, all users still use a particular shard chain, then the sharding fails and scalability has not been improved much. This is also a potential problem for ETH 2.0 as the ability to perform smart contract operations for each shard chain is not included in the initial planning.
More complicated blockchain explorers
The improvement of scalability brought by sharding is achieved by adding more algorithms and data to the whole network, greater processing power is needed for blockchain explorers to effectively retrieve information on the blockchain.

Other Blockchains Adopting Sharding

Elrond:

Elrond is a decentralized public blockchain aims to improve scalability, speed and security by developing three key technologies, including Adaptive State Sharding with blockchain scaling technology, Secure Proof of Stake (SPoS) which determines the selection of validators to speed up verification, and Elrond Virtual Machine, which supports multiple programming languages and is compatible with Ethereum VM.

Elrond’s unique Adaptive State Sharding combines three forms of sharding:

1. Network Sharding represents the process of grouping the nodes into shards.
2. Transaction Sharding takes the complexity to the next level and deals with the distribution of transactions across different shards, but all the nodes keep the entire blockchain into their state.
3. State sharding represents the most sophisticated part and is described as a mechanism that allows different shards to deal only with a portion of the state without replicating the data between nodes from different shards.

Currently, Elrond can reach 15,000 TPS while keeping the fee as low as $0.001, which is a reasonable advantage over Ethereum.

Near:

Near is a block public chain based on full-state sharding, simplified development applications and proof-of-stake mechanism. It sets the goal to increase scalability for decentralized application development and open the door of blockchain technology to everyone. Its Doomslug consensus mechanism,which adopts the Nightshade algorithm, is able to scale TPS to 100,000. Near’s technical architecture is different with other sharded public chains such as the Beacon Chain, which is composed of one chain and multiple shard chains, it is splitted into individual blocks, and sharding is performed between these blocks. A block contains all transactions of all shards while the shard state is divided, and validators are randomly assigned to verify the corresponding state of the shard of the transaction, thereby improving the security.

Zilliqa:

Zilliqa is a fairly early bird in blockchains. It was launched to solve the poor scalability of blockchain at that time by using network sharding and transaction sharding, as well as PoW and Byzantine fault-tolerant consensus algorithm (PBFT) with simplified complexity to accelerate consensus within shards. It can still run fast with more than 600 nodes, and constant scaling takes Zilliqa to higher TPS from 2400 to 3600 TPS.

Harmony:

Harmony is a public chain based on state sharding, which is structured by a Beacon Chain and multiple sharding chains and accompanied by the Byzantine fault-tolerant consensus algorithm (PBFT) to reach a deeply optimized consensus while speeding up with BLS signatures to aggregate multiple signatures into a single one. In a sharded public chain, 1% of the computing power can be used for double-spending attacks, regard this, Harmony adopts EPoS （Effective Proof-of-Stake）and Random Sharding to disperse large-scale staked tokens and randomly allocate them to multiple shards to reduce the risk of being attacked so as to enhance the security of shards. It also adopts Kademlia Cross-shard Communication to control the network expenditure while taking advantage of Erasure Code, which enables data recovery, to optimize the block broadcast process to perform efficient horizontal sharding scaling.

Conclusion

With the explosion of cryptocurrency usage and the proliferation of dAPPs, some traditional blockchains are overwhelmed and unable to meet the increasing market demand. How to improve the scalability without giving up decentralization and security has become a major issue for the current blockchain industry.

Sharding effectively diverts on-chain needs and improves access efficiency

Sharding is an optimization technology that improves database access efficiency by splitting on-chain transaction data and processing it by different nodes synchronously, this is how traffic diversion and speed increase are achieved. As a highly anticipated Layer 1 scaling solution, when sharding is successfully realized after overcoming the high-level technical difficulties, its potential benefits are quite huge. Sharding has been adopted by some blockchains and in the upcoming ETH 2.0, it will also be adopted with the hope to improve network performance significantly.

Sharding is extremely hard to achieve with many unsolved challenges, but eventually the impossible triangle will be overcome

However, sharding is not a perfect solution. In addition to the increased complexity, it may expose the network to more risks. A lot of developers are continuing their research on how to coordinate different shards to operate efficiently, expecting sharding to be the key to overcoming the “impossible triangle” of security, decentralization and scalability, so that all the existing blockchain applications and cryptocurrencies can be further popularized.

Author: Piccolo Translator:Yulei
Reviewer(s) : Hugo , Edward
Disclaimer:

• This article represents only the views of the observers and does not constitute any investment suggestions.
  *Gate.io reserves all rights to this article. Reposting of the article will be permitted provided Gate.io is referenced. In all other cases, legal action will be taken due to copyright infringement.