question

nyjdams_136971 avatar image
nyjdams_136971 asked vamsipm2020@yahoo.com commented

Why does starting DSE in the Academy course DS210 VM return "ulimit: error setting limit (Operation not permitted)"?

I'm trying to do the 'add a node' exercise in DS210.

So I've followed the instructions very carefully, but when I try to start cassandra, I get this:

 * Starting DSE daemon dse
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)
/etc/init.d/dse: 501: ulimit: error setting limit (Operation not permitted)
The linux limit 'memlock' is '64'. The minimum recommended is 'unlimited'. Check your configuration.
Check our documentation for more details. https://docs.datastax.com/en/dseRecommendedSettings68/
[ OK ]
ubuntu@ds210-node3:~$ ulimit -l
64
ubuntu@ds210-node3:~$

If I do 'ulimit -l' on node1, which is running cassandra quite happily, it's also 64.

Any ideas welcome please

academyds210
12 comments
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

nyjdams_136971 avatar image nyjdams_136971 commented ·

After trying a whole bunch of stuff and getting nowhere, I came back to node 1 which I'd said was running quite happily, and stopped it. But on trying to restart it, I got the same error. So I deleted the whole DS210 VM and started again. Very, very carefully, I set up node 1 as per the first exercise - but it fails with the exact same error. This is on a virgin, brand-new DS210 VM.

What is going on here?????

Any ideas would be very welcome.

0 Likes 0 ·
nyjdams_136971 avatar image nyjdams_136971 nyjdams_136971 commented ·

I have now tried deleting all VMs, uninstalling VirtualBox, clearing out trash, and restarting my computer. Then installed VirtualBox afresh, created VM DS210 afresh and again set up node1 even more carefully than before. The result is exactly the same as above. For what it's worth, the output from 'sudo apt-get install dse-full' includes this:

Setting up dse-libtomcat (6.8.0-1) ...
Setting up dse (6.8.0-1) ...
sysctl: setting key "vm.overcommit_memory": Read-only file system
sysctl: setting key "vm.max_map_count": Read-only file system
sysctl: setting key "net.ipv4.tcp_keepalive_time": Read-only file system
Setting up dse-libsolr (6.8.0-1) ...
Setting up dse-full (6.8.0-1) ...
Processing triggers for systemd (237-3ubuntu10.40) ...

But if I run that command again it says dse-full is up to date. So it looks as if the install succeeded, but whatever I try, I can't run cassandra, I still get the same error as listed in the original post.

0 Likes 0 ·
nyjdams_136971 avatar image nyjdams_136971 nyjdams_136971 commented ·

Just for the avoidance of doubt - a couple of days ago I got node 1 up and running successfully - this error is something that wasn't there two days ago.

0 Likes 0 ·
pacorulo avatar image pacorulo commented ·

Hi @nyjdams_136971 I had the same message and took me to remember things from the past...

First of all, I would like to say that C* process is up and running. You can check it with a ps or nodetool, so it is not something that makes us fail starting up C* and the cluster. I recommend you to not remove any VM again as it will be a waste of time :-)

[continued...]

0 Likes 0 ·
nyjdams_136971 avatar image nyjdams_136971 pacorulo commented ·

Thanks @pacorulo - the thing that puzzles me is that when I first set up this VM for the first exercise in DS210, it worked. That's what is really strange about all this.

0 Likes 0 ·
pacorulo avatar image pacorulo pacorulo commented ·

[continued...]

Checking the value of that parameter on the container (as I have realized the VM uses docker containers for each C* node on the ring we will create):

ubuntu@ds210-node1:~/labwork$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23064
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
0 Likes 0 ·
pacorulo avatar image pacorulo pacorulo commented ·

[continued...]

and it is 64 KB, as the message shows. The definition of the parameter (max locked memory) is: "The maximum size that may be locked into memory. Memory locking ensures the memory is always in RAM and never moved to the swap disk". So, it means some swap could be used by the container as also doing a docker inspect:

$ sudo docker inspect docker_node1_1 | grep -i swap
            "MemorySwap": -1,
            "MemorySwappiness": null,

we can see the container can use unlimited swap (that -1 tell it to us, as docker documentation explains) from the host.

0 Likes 0 ·
Show more comments
pacorulo avatar image pacorulo commented ·

I read the full output today and it gave me some more light as it points to the recommended settings needed to configure C*. Actually:

ubuntu@ds210-node1:~$ cat /etc/security/limits.d/cassandra.conf 
# Provided by the cassandra package
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited

so, as the C* process is run by cassandra user it will use this configuration.

[continued...]

0 Likes 0 ·
pacorulo avatar image pacorulo pacorulo commented ·

[continued...]

But even if I connect to the node, make cassandra able to run the service (with sudo rights and using ubuntu user), the message is exactly the same:

ubuntu@ds210-node1:~$ sudo -u cassandra sudo service dse status
[sudo] password for cassandra: 
 * dse is not running
ubuntu@ds210-node1:~$ sudo -u cassandra sudo service dse start 
 * Starting DSE daemon dse                                                                                                                                                                                                                             
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)
/etc/init.d/dse: 501: ulimit: error setting limit (Operation not permitted)
The linux limit 'memlock' is '64'. The minimum recommended is 'unlimited'. Check your configuration.
Check our documentation for more details. https://docs.datastax.com/en/dseRecommendedSettings68/
0 Likes 0 ·
pacorulo avatar image pacorulo pacorulo commented ·

[continued...]

My only doubt now is why this operation is working in this way and even if cassandra user executes the start command with sudo rights (inside the container) the message is still the same.

0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ commented ·

@nyjdams_136971 and @pacorulo, I apologise that we missed this question and took this long to respond. It somehow slipped through our radar and we didn't realise it hasn't been attended to. Please accept our apologies. Cheers!

0 Likes 0 ·

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered vamsipm2020@yahoo.com commented

Symptom

This error isn't a problem with the VM itself:

/etc/init.d/dse: 501: ulimit: error setting limit (Operation not permitted)

Background

The DS210 Cassandra Operations course at DataStax Academy requires three nodes in order to perform the exercises. The DS210 virtual machine included with the course runs 3 Docker instances:

$ sudo docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                  NAMES
951e2722387e        datastaxacademy/ds210:6.7   "/bin/sh -c '/etc/in…"   27 hours ago        Up 33 minutes       22/tcp, 7199/tcp, 8888/tcp, 9042/tcp   docker_node2_1
cbe717758482        datastaxacademy/ds210:6.7   "/bin/sh -c '/etc/in…"   27 hours ago        Up 33 minutes       22/tcp, 7199/tcp, 8888/tcp, 9042/tcp   docker_node3_1
d03b524f7747        datastaxacademy/ds210:6.7   "/bin/sh -c '/etc/in…"   27 hours ago        Up 33 minutes       22/tcp, 7199/tcp, 8888/tcp, 9042/tcp   docker_node1_1

Cause

Unfortunately, there were changes to Docker's security policies and setting ulimit from within the container is no longer allowed and can only be done on the host.

Otherwise, limits can only be set when the Docker container is run in --privileged mode.

Workaround

I don't have a fix to provide to you. You can instead try to run the exercises using your own VMs or on AWS EC2 instances. If you go down the EC2 route, there is information for it in the DS210 Exercise Environment Setup document. Cheers!

3 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

pacorulo avatar image pacorulo commented ·

@Erick Ramirez Hi Erick. Thanks for your reply and support. You do not have to apologize, swim in an ocean can be difficult on some occasions, I perfectly understand. It all made me think: running privileged container is not recommended, ulimit defined per user, containers running on the cloud... I will think about it all :-)

0 Likes 0 ·
shah_d_p_194172 avatar image shah_d_p_194172 commented ·

I am having my Oracle VM Virtual box manager on my Windows 10 laptop which has imported 'DS210-VM-6.7' in order to complete DS210 course (as per Datastax academy). However, I too am seeing the same above problem. But just to ensure that my understanding is correct - I wont be able to run DS210 related exercises on my laptop with my Oracle VM virtual box manager?

0 Likes 0 ·
vamsipm2020@yahoo.com avatar image vamsipm2020@yahoo.com commented ·

Will datastax academy be providing any solution to this problem soon? Also the DS210 exercise guide doesn't have instructions to use your own VMs. Any guidance on it? Is it like creating 3 new VMs from scratch (Like install Ubuntu & Other required SW) and just start the exercises?

0 Likes 0 ·