we have a 9 node spark cluster that is performing a background job with the help of Spark Job Server UI and Jenkins.
since last couple of days, I am observing significant time to complete the Spark job. However, I am sure we are not observing in new data load to the system. or not made any changes in job or anything.
One thing I would like to mention and for that I need help. most of the job which are taking longer time to complete is depend on table that has higher amount of tombstone.
- Average live cells per slice (last five minutes): 9.260071466283065
- Maximum live cells per slice (last five minutes): 5001.0
- Average tombstones per slice (last five minutes): 1.3884200136199125
- Maximum tombstones per slice (last five minutes): 9795.0
I know that having higher amount of tombstone may cause read issue while perroming background operation. version of DSE is quite old 4.8.* and the table contains DateTier Compation so I am not sure how can I bring down this tombstone as mannual compation or read reapir is not good with datetier (Correct me If I am wrong here)