Spark streaming gets stucked (cassandra-connector V3.0.0)

Question

question

tomas.bartalos_162914 asked · Oct 02, 2020 at 03:39 PM

Spark streaming gets stucked (cassandra-connector V3.0.0)

Hello,

I have 3 Spark streams writing data to cassandra using spark-connector 3.0.0-beta. After 108324 completed tasks, the streams got stucked forever:

This is pretty dangerous behavior, because after huge amount of successful tasks the stream just stopped writing data, without any errors. Cassandra DB is in healthy state, all nodes UP.

My cassandra DB version is: 3.11.5

Driver thread dump: thread_dump_driver.txt

Executor thread dump: thread_dump_cassandra.pdf

All worker threads are doing:

CassandraConnector.closeResourceAfterUsed -> Session.refreshSchema

Is it possible to specify timeout for operations like refreshSchema ?

Recently I've reported issue with stucked streams which happened on stream start. But this one happened after lot of completed tasks.

spark-cassandra-connector

screenshot-2020-10-02-at-165027.png (124.4 KiB)

screenshot-2020-10-02-at-171150.png (107.5 KiB)

thread-dump-driver.txt (18.6 KiB)

thread-dump-cassandra.pdf (155.5 KiB)

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Answer 1 · 2020-10-06T06:23:41Z

jaroslaw.grabowski_50515 answered · Oct 06, 2020 at 06:23 AM

Hi @tomas.bartalos_162914,

Could you please provide a repro steps for this problem?

Do executor threads await for a schema refresh indefinitely? Is it possible that these particular threads moved on after the stack trace was generated?

If the executor threads are stuck forever I'd expect an exception in db/executors logs, could you verify it?

It is possible to set the timeout for Schema queries with this setting but if there is a deadlock somewhere the setting won't do much. The default timeout is 5 seconds.

5 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

tomas.bartalos_162914 · Oct 07, 2020 at 10:57 AM

Hello, thank you for your reply. The executor threads were for sure waiting indefinitely (average job time is 1-10 seconds and they've been waiting for 47 hours). The threads were not moving, I made several dumps with the same stack results.

We're running Cassandra DB on Kubernetes (bitnami helm chart) and I've found out that at the moment when stream got stuck, all containers of Cassandra DB was killed by k8s, and afterwards immediately recreated.

There is no exception in Spark driver logs.

0 ·

jaroslaw.grabowski_50515 tomas.bartalos_162914 · Oct 07, 2020 at 11:12 AM

Could you please create a minimal reproduction for this problem?

0 ·

tomas.bartalos_162914 jaroslaw.grabowski_50515 · Oct 07, 2020 at 12:54 PM

I don't think I can provide a minimal code to reproduce this. The behavior is non-deterministic and depends on timing. According to my investigation to reproduce steps to reproduce:

start spark streaming to cassandra

df.write.format("org.apache.spark.sql.cassandra").save()

kill cassandra nodes in the right time

0 ·

Show more comments

Answer 2 · 2020-10-05T01:16:57Z

Erick Ramirez answered · Oct 05, 2020 at 01:16 AM

Thanks for sticking with us. I suspect this is related to multiple issues being addressed by SPARKC-614.

Jarek Grabowski (@jaroslaw.grabowski_50515) has been working on this problem for a couple of weeks now. Spark-Cassandra connector v3.0.0 was released a few days ago and I think it will address the issues you reported.

Please try it out and let us know how you go. Thanks again. Cheers!

2 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

tomas.bartalos_162914 · Oct 05, 2020 at 06:31 PM

Thank you for your reply, I will switch to stable V3.0.0. However I don't think SPARKC-614 is related to this issue. It solves many objects creation perf problem (changing def -> val), but this is about threads not being notified and stuck forever.

0 ·

tomas.bartalos_162914 tomas.bartalos_162914 · Dec 14, 2020 at 10:31 AM

Hello,

I'd like to emphasize this is happening also with stable V3.0.0 connector. It happens ~ once in a month. Each executor thread is stuck on

Session.refreshSchema()

I'm attaching stack trace and screenshot of stucked jobs (notice the duration column)

stack-trace.txt

0 ·

screenshot-2020-12-14-at-112618.png (222.8 KiB)

stack-trace.txt (3.3 KiB)

question