Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

ted.petersson_164115 avatar image
ted.petersson_164115 asked Erick Ramirez edited

What are the image requirements when setting a custom serverImage with cass-operator?

I want to run my custom Cassandra image using the serverImage parameter from the CRD:
https://github.com/datastax/cass-operator/tree/master/docs/user#specifying-version-and-image

I just tested by setting it like this (by selecting the standard Cassandra docker image):

  serverType: cassandra
  serverVersion: "3.11.6"
  serverImage: "cassandra:3.11"

But that did not work - the Pod could not start the container properly...

So I guess there are some special requirements on the serverImage specified here?

When I deploy without giving the serverImage parameter the Pod starts with the following cassandra image:

image: datastax/cassandra-mgmtapi-3_11_6:v0.1.2

Looking into that Dockerfile :
https://github.com/datastax/management-api-for-apache-cassandra/blob/master/Dockerfile-3_11
It seems to be a multi-stage build with some "extra management stuff" added to the cassandra:3.11 image...

I guess that's how the cass-operator "manages" Cassandra? That functionality is built into the server (Cassandra) image?

When looking at the documentation: https://github.com/datastax/management-api-for-apache-cassandra
It says "The Management API is a sidecar service layer...". So I first imagined that it would be a sidecar container inside the Pod. But it seems to be "part of" the server image... Am I right?

Because the only other (by default) container in the Pod seems to be the "server-system-logger".

So back to the primary question - how can I run my custom Cassandra image with cass-operator?

Do my custom image have to include the management-api "stuff" for it to work? Similar to how the Dockerfile-3_11 (referred to above) does it?
Or is there any smoother way to do it?

cass-operator
1 comment
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ ·

I've reached out to the team internally to get you an answer. Cheers!

0 Likes 0 ·

2 Answers

jim.dickinson_187342 avatar image
jim.dickinson_187342 answered jim.dickinson_187342 commented

Cass Operator requires using the management-api-for-apache-cassandra REST interface. You're welcome to take the docker images and add anything to them. Do you mind elaborating a little on how you've customized the "official" Cassandra image? (I'm quoting it because it's not built by anyone affiliated with the Apache Cassandra project AFAIK, just some hard working Docker developers - https://github.com/docker-library/cassandra)

4 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

ted.petersson_164115 avatar image ted.petersson_164115 ·

There are 2 major reasons why we want to have a "custom Cassandra image".
* We have a custom Cassandra build (with some minor 4.0 features backported to 3.11.6)
* We want to have different base os for the image.

So this custom Cassandra image differs somewhat from the docker-library version of Cassandra...

So my questions are:

1 - Is it even possible to deploy this custom image with cass-operator?

2 - What are the requirements on the serverImage image? Are they specified somewhere?

3 - What would be the least effort to make this happen?

4 - Is it possible to run the Management API as a sidecar instead of running it "inside" the Cassandra container? The idea would the be to have (if I understand this correctly)
* serverImage (my custom cassandra imgare) running as the "cassandra" container
* management-api REST interface running as a sidecar


0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ ted.petersson_164115 ·

1 - Yes, you can deploy a custom image with the cass-operator. You didn't specify in your original post why the pod couldn't start it. What was the result of your investigation?

2 - Nothing specific. The root cause of the failure in your attempt will give us clues.

3 - See 1 & 2. Your answer to Jim Dickinson's question on "how you customised" the image is crucial.

4 - It runs as a sidecar service but maybe I misunderstood what you mean. I might have to defer this to Jim.

Cheers!

0 Likes 0 ·
jim.dickinson_187342 avatar image jim.dickinson_187342 ted.petersson_164115 ·

1 - Yes, it's possible to make significant changes to the image and to still use it with Cass Operator.

2 - We need to write up a doc about what's required. Maybe working with you could help us get the base of this doc setup?

3 - See below.

4 - No. The Management API brings up Cassandra by shelling out and using the Cassandra shell scripts. It also uses a unix socket file to send CQL to the server - no JMX is used.


I think you should check out how we build the Docker images - it should be pretty easy for you to make changes if you've already got a working Docker image with your changes.

https://github.com/datastax/management-api-for-apache-cassandra/blob/master/.github/workflows/ci.yaml#L22

docker build -t management-api-for-apache-cassandra-builder -f ./Dockerfile-build ./
docker build -t mgmtapi-3_11 -f Dockerfile-3_11 .


So first we compile the Management API here...

https://github.com/datastax/management-api-for-apache-cassandra/blob/master/Dockerfile-build

0 Likes 0 ·
jim.dickinson_187342 avatar image jim.dickinson_187342 jim.dickinson_187342 ·

continued...


Then we get those compiled files into the upstream Cassandra 3.11 here...

https://github.com/datastax/management-api-for-apache-cassandra/blob/master/Dockerfile-3_11


Can you try this out? Thanks!

0 Likes 0 ·
rcosnita avatar image
rcosnita answered john.sanda_194109 commented

I stumbled upon a similar problem and the comments from this thread help a lot. In my case, I had to build a custom image which contained also the lucene index and medusa.

You can see a working example of how the Dockerfile looks like here: https://github.com/rcosnita/cassandra-lucene-index/blob/master/Dockerfile

You can also use the final docker image from docker hub: rcosnita/cassandra-lucene-index:3.11.6.1

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Erick Ramirez avatar image Erick Ramirez ♦♦ ·

Thanks so much for sharing this! I'm sure @john.sanda_194109 would be interested in your experience with Medusa + k8s.

Would you be interested in collaborating with us to publish your experience in setting this up? It could be as simple as a blog post. We'd be happy to send you some swag if that entices you. Cheers!

1 Like 1 ·
john.sanda_194109 avatar image john.sanda_194109 ·

Very cool! I would love to hear more about your experiences with Medusa in k8s :)

I am working on k8s integration for Medusa which involves exposing a gRPC service for backups that will run in a separate sidecar container. There are some details about it here.

I am also working on medusa-operator which includes a backup controller.

The plan is to eventually add support in Cass Operator as well.

0 Likes 0 ·