Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

nyjdams_136971 avatar image
nyjdams_136971 asked ·

DS210 exercise - ulimit should be 'unlimited'

I'm trying to do the 'add a node' exercise in DS210.

So I've followed the instructions very carefully, but when I try to start cassandra, I get this:

* Starting DSE daemon dse
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)
/etc/init.d/dse: 501: ulimit: error setting limit (Operation not permitted)
The linux limit 'memlock' is '64'. The minimum recommended is 'unlimited'. Check your configuration.
Check our documentation for more details. https://docs.datastax.com/en/dseRecommendedSettings68/
[ OK ]
ubuntu@ds210-node3:~$ ulimit -l
64
ubuntu@ds210-node3:~$


If I do 'ulimit -l' on node1, which is running cassandra quite happily, it's also 64.

Any ideas welcome please

datastax cassandra
3 comments
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

After trying a whole bunch of stuff and getting nowhere, I came back to node 1 which I'd said was running quite happily, and stopped it. But on trying to restart it, I got the same error. So I deleted the whole DS210 VM and started again. Very, very carefully, I set up node 1 as per the first exercise - but it fails with the exact same error. This is on a virgin, brand-new DS210 VM.

What is going on here?????

Any ideas would be very welcome.

0 Likes 0 · ·
nyjdams_136971 avatar image nyjdams_136971 nyjdams_136971 ·

I have now tried deleting all VMs, uninstalling VirtualBox, clearing out trash, and restarting my computer. Then installed VirtualBox afresh, created VM DS210 afresh and again set up node1 even more carefully than before. The result is exactly the same as above. For what it's worth, the output from 'sudo apt-get install dse-full' includes this:

Setting up dse-libtomcat (6.8.0-1) ...
Setting up dse (6.8.0-1) ...
sysctl: setting key "vm.overcommit_memory": Read-only file system
sysctl: setting key "vm.max_map_count": Read-only file system
sysctl: setting key "net.ipv4.tcp_keepalive_time": Read-only file system
Setting up dse-libsolr (6.8.0-1) ...
Setting up dse-full (6.8.0-1) ...
Processing triggers for systemd (237-3ubuntu10.40) ...

But if I run that command again it says dse-full is up to date. So it looks as if the install succeeded, but whatever I try, I can't run cassandra, I still get the same error as listed in the original post.

0 Likes 0 · ·
nyjdams_136971 avatar image nyjdams_136971 nyjdams_136971 ·

Just for the avoidance of doubt - a couple of days ago I got node 1 up and running successfully - this error is something that wasn't there two days ago.

0 Likes 0 · ·
pacorulo avatar image
pacorulo answered ·

I read the full output today and it gave me some more light as it points to the recommended settings needed to configure C*. Actually:

ubuntu@ds210-node1:~$ cat /etc/security/limits.d/cassandra.conf 
# Provided by the cassandra package
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited

so, as the C* process is run by cassandra user it will use this configuration.


But even if I connect to the node, make cassandra able to run the service (with sudo rights and using ubuntu user), the message is exactly the same:

ubuntu@ds210-node1:~$ sudo -u cassandra sudo service dse status
[sudo] password for cassandra: 
 * dse is not running
ubuntu@ds210-node1:~$ sudo -u cassandra sudo service dse start 
 * Starting DSE daemon dse                                                                                                                                                                                                                             
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)
/etc/init.d/dse: 501: ulimit: error setting limit (Operation not permitted)
The linux limit 'memlock' is '64'. The minimum recommended is 'unlimited'. Check your configuration.
Check our documentation for more details. https://docs.datastax.com/en/dseRecommendedSettings68/

My only doubt now is why this operation is working in this way and even if cassandra user executes the start command with sudo rights (inside the container) the message is still the same.

Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pacorulo avatar image
pacorulo answered ·

Hi @nyjdams_136971 I had the same message and took me to remember things from the past...

First of all, I would like to say that C* process is up and running. You can check it with a ps or nodetool, so it is not something that makes us fail starting up C* and the cluster. I recommend you to not remove any VM again as it will be a waste of time :-)

Checking the value of that parameter on the container (as I have realized the VM uses docker containers for each C* node on the ring we will create):

ubuntu@ds210-node1:~/labwork$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23064
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and it is 64 KB, as the message shows. The definition of the parameter (max locked memory) is: "The maximum size that may be locked into memory. Memory locking ensures the memory is always in RAM and never moved to the swap disk". So, it means some swap could be used by the container as also doing a docker inspect:

$ sudo docker inspect docker_node1_1 | grep -i swap
            "MemorySwap": -1,
            "MemorySwappiness": null,

we can see the container can use unlimited swap (that -1 tell it to us, as docker documentation explains) from the host. It is supposed C* must not use swap but probably we will not have problems on our exercises as the container is using 1.75Gb of RAM:

$ sudo docker stats docker_node1_1

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
f4d8bd81eb1c        docker_node1_1      56.77%              1.716GiB / 1.75GiB   98.08%              1.25GB / 23.3MB     1.58GB / 126MB      93


Making a recap (as i wrote too much information), I think we do not have to worry. But I wrote down all that info as maybe some Datastax C* expert could give us some more light on it all :-)

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Thanks @pacorulo - the thing that puzzles me is that when I first set up this VM for the first exercise in DS210, it worked. That's what is really strange about all this.

0 Likes 0 · ·