question

You should be able to meet your customer's requirements with nodetool snapshot. And the operation is a lot simpler since you stated in (1) that there won't be application traffic.

For what it's worth, it isn't necessary to stand-up another DC in (2) just to do backups. I should also point out that EBS volume snapshots you've mentioned in (4) is not a recommended way of backing up Cassandra since (a) not all the data files would be consistent and (b) data in memtables would not be included in the backup (until they're flushed to disk which C* snapshots do).

Example backup

Here is an example where I take a snapshot of the community keyspace.

STEP B1 - Create a snapshot:

$ nodetool snapshot community
Requested creating snapshot(s) for [community] with snapshot name [1591083719993] and options {skipFlush=false}
Snapshot directory: 1591083719993

The directory 1591083719993 name is a Unix timestamp for when the snapshot was created and is equivalent to June 2, 2020 7:41am GMT. There is one table called users in my example keyspace and the snapshot is located in the following directory structure:

data/
  community/
    users-6140f420a4a411ea9212efde68e7dd4b/
      snapshots/
        1591083719993/
          manifest.json
          mc-1-big-CompressionInfo.db
          mc-1-big-Data.db
          mc-1-big-Digest.crc32
          mc-1-big-Filter.db
          mc-1-big-Index.db
          mc-1-big-Statistics.db
          mc-1-big-Summary.db
          mc-1-big-TOC.txt
          schema.cql

Example restore

Here is an example where I restore the table.

PREPARATION - Shutdown application(s) to temporarily stop traffic to the cluster. Clear out the data from all the target tables. This command I used on my table:

cqlsh> TRUNCATE TABLE community.users;

STEP R1 - To restore a table, copy the data files in the snapshot subfolder to the table's directory. Using my table's snapshot example above:

$ cd users-6140f420a4a411ea9212efde68e7dd4b
$ cp -p snapshots/1591083719993/*.db .

STEP R2 - Force C* to reload the SSTables on disk:

$ nodetool refresh community users

Check the debug.log on the node to confirm that the SSTables were loaded.

STEP R3 - Repeat steps R1-R2 until all tables have been restored.

Tooling

Both the backup and restore examples I posted only does the job on one node. With the exception of TRUNCATE, all the steps need to be carried out on all nodes in the cluster.

To achieve this, I recommend using tools you already have in your environment. If you are already using orchestration tools like Ansible, create the snapshots in parallel by running the command on all nodes simultaneously. Similarly, you can also script the restore operation so you can execute it in parallel using Ansible.

If you are not using orchestration tools, consider using Cluster SSH (cssh) or Parallel SSH (pssh) so you can run commands simultaneously on all nodes in your cluster.

Backup archive

I realise your customer does not have a requirement for off-server backups but I'll mention this anyway for completeness. For open-source Apache Cassandra, we recommend using Medusa backup software maintained by the team at The Last Pickle (who coincidentally joined Datastax in early 2020).

Medusa provides a facility for archiving backups to NFS via a local mount point as well as off-server locations including AWS S3 and Google Cloud Storage. It also makes it easy to perform restores on the same cluster or another remote cluster (cloning) from off-server backup archives.

For more info, see Backing up and restoring data. Cheers!