Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

randas avatar image
randas asked Erick Ramirez answered

Is there a way to aggregate all Spark stdout, stderr, logs into one?

I've spark 3.0.2 cluster in stand alone mode.
I need to aggregate all the nodes logs that we can see in stdout or stderr in one location.
I could see some way to do this in yarn but not in stand alone.
Is there any way instead of using an ELK Stack ?

spark
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered

When you combine all logs/outputs into one means it would be difficult to determine where each individual message/line came from. What you're asking doesn't make sense to me so perhaps I misunderstood your intent.

In any case, this is very specific to Spark so I'd recommend asking in a Spark forum. Cheers!

Share
10 |1000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.