question

ranjeet_ranjee avatar image
ranjeet_ranjee asked Erick Ramirez answered

How do I change the commitlog_directory location on a production cluster?

Hi,

Please suggest to me How to change commitlog_directory location on the production Cluster and the ideal recommended disk size for the commitlog_directory.

Here you go key information

1) 3 Node Cluster
2) RF -2
3) DSE Version - 6.8.4

Disk Storage

[root@cassandra-11 commitlog]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G   17M   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vda1       640G  286G  355G  45% /
tmpfs           3.2G     0  3.2G   0% /run/user/997
tmpfs           3.2G     0  3.2G   0% /run/user/0

Nodetool status

[root@cassandra-11 commitlog]# nodetool status
Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving/Stopped
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.131.3.66  275.78 GiB  1            66.4%             81153fda-3bd4-4272-95ac-3a4a196bcaf2  rack1
UN  10.131.3.07  260.22 GiB  1            79.9%             083f050d-2c4b-41e3-b2f6-3c8b16234274  rack1
UN  10.131.3.48  510.69 GiB  1            53.7%             fb9f7d50-2e86-4ba7-9798-9316d1023bde  rack1
commitlog
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

Here are the steps to change the commitlog directory location.

Preparation

Change the directory by updating the following line in cassandra.yaml:

commitlog_directory: /full/path/to/commitlog

NOTE: This change will not take effect until DSE is restarted.

Implementation

Perform a rolling restart with this procedure:

Step 1 - On the first node, force the memtables to be flushed to disk so there are no commitlogs to replay on startup with the following command:

$ nodetool drain

Step 2 - Shutdown DSE.

Step 3 - Verify from the logs that DSE shutdown cleanly. In a separate SSH session, run:

$ tail -100f /path/to/system.log

Keep this session running so you can monitor the startup.

Step 4 - Start DSE.

Step 5 - Monitor the progress of the startup by watching the logs.

Step 6 - Verify that DSE is operational.

$ netstat -tlnp
$ sudo lsof -nPi -sTCP:LISTEN
$ nodetool status

Step 7 - Repeat steps 1 to 6 above on the next node in the cluster.

Step 8 - Repeat steps above until all nodes in the cluster have been restarted.

Volume size

You will need a minimum of 8GB but 16 or 32GB would be ideal. Cheers!

Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.