Using the nodetool Utility

The nodetool utility is a command-line interface for managing a cluster.

• A command-line interface for monitoring Cassandra
• Also used for performing routine database operations
• Run directly from an operating Cassandra node

Note: If the node from which you issue the command is the intended target, you do not need the -h option to identify the target; otherwise, for remote invocation, identify the target node, or nodes, using -h.

How does it work?:

• JMX(Java Management Extention) command line wrapper.
• Communicates with JMX to perform Operation and Monitoring tasks exposed by MBeans
• JMX is a Java technology that supplies tools for managing and monitoring Java applications and services.

Where does it work on? :

The nodetool commands can be categorized into these groups: 1) Cluster 2) Server 3) Backup 4) Storage 5) Compaction 6) Network


nodetool status:

Type of commands: Cluster

• This has to do with working with the cluster-wide information
• As the point-of-view of that node as it sees the state of the cluster.

1) nodetool status

• Provides information about the cluster, such as the state, load, and IDs.
• Who's up and who's down.
• This is probably the most used nodetool command.

$ nodetool <options> status <keyspace>

2) nodetool repair

• Starts a repair process from the point of the view of that node.
• Repairs one or more keyspaces, tables.

$ nodetool <options> repair

Type of commands: Node

• Scope of the commands are at the node level.

1) nodetool info

• Provides node information such as load and uptime
• Status of the JVM
• Use the "T" flag to display all tokens

$ nodetool <options> info ( -T | --tokens )

2) nodetool tpstats

• Provides usage statistics of thread pools
• How many completed, how many pending, which ones are blocked.
• A high number of pending tasks for any pool can indicate performance problems
• Shows mutation drops( critical for troubleshooting a node)
• A heavily used command during troubleshooting

$ nodetool <options> tpstats

Statuses: Active, Pending, Completed, Blocked, All time blocked, Pool Names : ReadStage RequestReleaseStage MutationStage ReadRepairStage GossipStage Anti-EntropyStage MigrationStage MemoryMeter MemtablePostFlusher FlushWriter Commitlog_Archiver InternalResponseStage HintedHandoff Message Type: Type of operations get dropped Range_Slice Read_Repair Paged_Range Binary Read Mutation _Trace Request_Response Counter_Mutation
3) nodetool tablehistograms

• The nodetool tablehistograms command provides statistics about a table
• Includes read/write latency, partition size, column count, and number of SSTABLES
• The report is incremental, not cumulative.
• It covers all the operation since the last time nodetool tablehistograms was run in the current session.
• The use of the metrics-core library makes the output more informative and easier to understand
• These statistics could be used to plot a frequency function.

$ nodetool <options> tablehistograms -- <keyspace>.<table>

Type of commands: Backup

1) nodetool snapshot

• We can take a snapshot of one or more keyspaces, or of a table, to backup data.
• Cassandra flushes the node before taking the snapshot.
• Takes the snapshot and store the data in the snapshot directory of each keyspace in the data directory.
• If you do not specify the name of the snapshot directory using the -T option, Cassandra names the directory using the timestamp of the snapshot like 398276394348934.

$ nodetool <options> snapshot Example: Snapshot all keyspaces $ nodetool snapshot Example: Snapshot single keypsace $ nodetool snapshot -t 2016.05.17 killrvideo Example: Multiple keyspace snapshot $ nodetool snapshot mykeyspace killrvideo Example: Single table snapshot $ nodetool snapshot --table users killrvideo Example: Snapshot different tables from different keyspaces $ nodetool snapshot -kt Cycling.Cyclist_Name, KillrVideo.Users

2) nodetool clearsnapshot:

• Removes one or more snapshots.

$ nodetool <options> clearsnapshot --t snapshot --keyspace Keyspace_Name

3) nodetool listsnapshots:

• Lists snapshot names, size on disk, and true size.

  $ nodetool <options> listsnapshots

Type of commands: Storage
1) nodetool cleanup:

• Used to get rid of old data on nodes after bootstrap operations.

$ nodetool <options> cleanup -- <keyspace> (<table> ...)

2) nodetool flush:

• The command flushes everything in the memtables out to SSTables and deletes all commitlog segments.
• You can specify a keyspace followed by one or more tables that you want to flush from the memtable to SSTables on disk.

$ nodetool <options> flush -- <keyspace> ( <table> ... )

Type of commands: Compaction 1) nodetool compact:

• Forces a major compaction on one or more tables for size tired compaction.
• Acts differently for different compaction strategies.

$ nodetool <options> compact <keyspace> (<table> ...)

2) nodetool compactionstats:

• Provides statistics about a compaction
• Not something you would just run all the time.
• This is a JMX statistic you could also pull in using OpsCenter
• See what is currently compacting

$ nodetool <options> compactionstats -H

* H converts bytes to human-readable form: KB, MB, GB, or TB

Type of commands: Network

1) nodetool proxyhistograms:

• Provides a histogram of network statistics
• The output of this command shows the full request latency recorded by the coordinator.
• Includes the percentile rank of the read and write latency values for inter-node communication
• Typically, you use the command to see if requests encounter a slow node.

       $ nodetool <options> proxyhistograms

2) nodetool netstats:

• Provides network information about host
• The default host is the connect host if the user does not include a hostname or ip-address in the command.

$ nodetool <options> netstats -H

Removing a node from cluster.

Why do we remove nodes from cluster:

• To reduce the capacity of the cluster.
• Decommission due to operational requirements eg: dse upgrade, swapping instances...
• Node is offline and will never come back online. eg: hardware issues...

While the node being removed:

• Other nodes need to pick up the removed node's data.
• The cluster needs to know the node is gone. 

3 options to remove a node from the cluster, based on the context:

• nodetool decommission
• nodetool removenode
• nodetool assassinate

Decommission: Removing a live node from the cluster. 

Decommissioning a node will assign the ranges of the old node to other nodes of the cluster. That node data will gets streamed evenly among the other active nodes.

Decommissioning will transfer the data from the decommissioned node to other active nodes in the cluster. With Vnodes, the rebalance happens automatically.

After running the nodetool decommission command:

• The node is offline i.e. node will not be shown in nodetool status.
• The dse services will still be running. Run "sudo services dse stop" to stop the services.
• The data will not be deleted from the decommissioned node.
• Not deleting the data may cause data resurrection issues.

$ nodetool decommission 

Note: Monitor decommission progress with nodetool netstats and nodetool status.

Streaming data to active nodes

                    

Removenode: Used to remove a dead down node from the cluster. 

Check the status of the node being removed through nodetool status. If the node is down and no chance of coming back, remove the node using the nodetool removenode command.

Removing a dead node from the cluster is done to reassign the token ranges that the dead node was responsible for to other nodes in the cluster and to populate other nodes with the data that the dead node had been responsible for. This process is initiated by nodetool removenode command.

With a dead node being dead, the data that it was responsible for needs to come from other nodes in the cluster, which happens when the nodetool removenode command is run.

We can check the node removal status using nodetool removenode status command.

$ nodetool removenode $host-id

Note: host-id can be obtained from nodetool status.



Assassinate:

The nodetool assassinate command is primarily intended to eliminate the problematic nodes from the cluster after the nodetool decommission or nodetool removenode commands have been executed. 

Do this as a last resort if the node is offline and never come back. It forcefully removes a dead node without re-replicating any data. It makes the remaining nodes in the cluster, aware that the node is gone.

Once the assassinated node marked as left, the node's token ranges will be owned by the other active nodes. Run nodetool repair on the remaining nodes to fix the data replication.

$ nodetool assassinate $ip-address