Removing a node from cluster.

Why do we remove nodes from cluster:

• To reduce the capacity of the cluster.
• Decommission due to operational requirements eg: dse upgrade, swapping instances...
• Node is offline and will never come back online. eg: hardware issues...

While the node being removed:

• Other nodes need to pick up the removed node's data.
• The cluster needs to know the node is gone. 

3 options to remove a node from the cluster, based on the context:

• nodetool decommission
• nodetool removenode
• nodetool assassinate

Decommission: Removing a live node from the cluster. 

Decommissioning a node will assign the ranges of the old node to other nodes of the cluster. That node data will gets streamed evenly among the other active nodes.

Decommissioning will transfer the data from the decommissioned node to other active nodes in the cluster. With Vnodes, the rebalance happens automatically.

After running the nodetool decommission command:

• The node is offline i.e. node will not be shown in nodetool status.
• The dse services will still be running. Run "sudo services dse stop" to stop the services.
• The data will not be deleted from the decommissioned node.
• Not deleting the data may cause data resurrection issues.

$ nodetool decommission 

Note: Monitor decommission progress with nodetool netstats and nodetool status.

Streaming data to active nodes

                    

Removenode: Used to remove a dead down node from the cluster. 

Check the status of the node being removed through nodetool status. If the node is down and no chance of coming back, remove the node using the nodetool removenode command.

Removing a dead node from the cluster is done to reassign the token ranges that the dead node was responsible for to other nodes in the cluster and to populate other nodes with the data that the dead node had been responsible for. This process is initiated by nodetool removenode command.

With a dead node being dead, the data that it was responsible for needs to come from other nodes in the cluster, which happens when the nodetool removenode command is run.

We can check the node removal status using nodetool removenode status command.

$ nodetool removenode $host-id

Note: host-id can be obtained from nodetool status.



Assassinate:

The nodetool assassinate command is primarily intended to eliminate the problematic nodes from the cluster after the nodetool decommission or nodetool removenode commands have been executed. 

Do this as a last resort if the node is offline and never come back. It forcefully removes a dead node without re-replicating any data. It makes the remaining nodes in the cluster, aware that the node is gone.

Once the assassinated node marked as left, the node's token ranges will be owned by the other active nodes. Run nodetool repair on the remaining nodes to fix the data replication.

$ nodetool assassinate $ip-address