Bringing together the Apache Cassandra experts from the community and DataStax.

Want to learn? Have a question? Want to share your expertise? You are in the right place!

Not sure where to begin? Getting Started

 

question

andrei_192823 avatar image
andrei_192823 asked ·

How can we fix node.js "DriverError: Socket was closed" when connecting to AWS Keyspaces DB?

We are using node.js serverless solution for our application and AWS Keyspaces with a single-region strategy as the main database. Cassandra-driver is used as a node.js driver.

For some reason we are getting ERROR NoHostAvailableError: All host(s) tried for query failed. First host tried, {IP:}9142: DriverError: Socket was closed pretty often, but absolutely randomly.

Currently, we have near 20 handlers, each of them uses its own connection(due to the serverless nature). Connect function described below:

const getClient = () => {
  const authProvider = new PlainTextAuthProvider(IAM_USER, IAM_PW);
  const certFilePath = path.join(__dirname, 'AmazonRootCA1.pem');
 
  const sslOptions = {
    cert: fs.readFileSync(certFilePath),
    host: AUTH_HOST,
    rejectUnauthorized: true,
  };
 
  const clientOptions = {
    contactPoints: CONTACT_POINTS,
    localDataCenter: LOCAL_DC,
    keyspace: KEYSPACE,
 
    authProvider: authProvider,
    sslOptions: sslOptions,
    pooling: { heartBeatInterval: 0, warmup: false },
 
    protocolOptions: {
      port: 9142,
    },
    queryOptions: { consistency: types.consistencies.localQuorum },
  };
 
  const client = new Client(clientOptions);
 
  if (IS_ENABLE_DEBUG_LOGGING) {
    client.on('log', (level, className, message) => {
      if (level !== 'verbose') {
        console.log('Cassandra driver log event:', level, className, message);
      }
    });
  }
 
  return client;
};
 
let cachedClient = null;
 
const connect = async () => {
  if (!cachedClient) {
    console.log('DB optimization: first connect');
  }
 
  if (cachedClient) {
    console.log(
      'DB optimization: connect exists, connected hosts',
      cachedClient.getState().getConnectedHosts().length,
      cachedClient.getState().getConnectedHosts(),
      cachedClient.getState(),
    );
  }
 
  if (cachedClient && cachedClient.getState().getConnectedHosts().length === 0) {
    console.log('DB optimization: connect exists, but connected hosts array empty, need reconnect');
  }
 
  if (cachedClient && cachedClient.getState().getConnectedHosts().length > 0) {
    console.log('DB optimization: connect exists and connected hosts array more than 0');
 
    return cachedClient;
  }
 
  try {
    const client = getClient(options, withoutKeyspace);
 
    await client.connect();
 
    console.log('Connected to the DSE cluster, discovered %d nodes', client.hosts.length);
 
    cachedClient = client;
 
    return client;
  } catch (error) {
    console.error('There was an error trying to connect', error);
    throw error;
  }
};
 
module.exports = {
  connect,
};

We are caching connection for each handler because initial Keyspaces connect can take near 2-6 seconds, so it does not make sense to open(and then close) a new connection per each request.

Currently, we have only one keyspace, but we are planning to separate tables to a different keyspace by their domains, but we don't think that this is the root cause.

Any suggestions on how to fix this issue?

Thanks!

aws keyspacesnode.js driver
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered ·

To my knowledge (and I'm happy to be corrected), we don't specifically test the drivers against AWS Keyspaces but we'll do our best to assist you.

I've noted that you've disabled heartbeat and I wonder if Keyspaces is closing the socket when it thinks the connection is idle:

    pooling: { heartBeatInterval: 0, warmup: false },

Have you logged a ticket with AWS in case this is a known issue? In the meantime, I'm going to reach out internally to the Drivers team at DataStax to get you some help. Cheers!

5 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Hi Erick,

We've been trying to use it without disabling heartBeatInterval, but got the same error. We've added those setting when found this example https://github.com/DataStax-Examples/aws-lambda-nodejs/blob/master/handler.js.

And yes, I've also logged a ticket with AWS, but there was no answers yet. At least we are trying to understand is it somehow related to Keyspaces implementation or node driver.

Thanks!

0 Likes 0 · ·

Disabling the heartbeat on the client side should prevent that behaviour but it looks to me the driver error is the outcome, not the cause. What I mean is that the socket is being closed by Keyspaces and the driver is just reporting it in the logs.

It would be good to get the response from AWS as they might be able to provide us clues as to what's going on. Cheers!

0 Likes 0 · ·

Just wondering if you had any luck with AWS. It would be good to understand what is causing the problem. Cheers!

0 Likes 0 · ·
andrei_192823 avatar image andrei_192823 Erick Ramirez ♦♦ ·

Unfortunately no. We'll try to contact their business support in a couple of days. I'll keep you updated as soon as we get any info from AWS.

[Follow up question posted as #6565]

0 Likes 0 · ·
Show more comments