question

Sun_P avatar image
Sun_P asked Erick Ramirez commented

What are the options for migrating data to a new table with a new schema?

If we want to migrate data from old table to new table (new table with schema change) then what are options available ?

old table:-

CREATE TABLE old_table (
  created_date timestamp,
  track_id text,
  name text,
  PRIMARY KEY (created_date)
)

New Table:-

CREATE TABLE new_table (
  created_date timestamp,
  week number,
  message text,
  track_id text,
  name text,
  PRIMARY KEY ( (created_date, week) )
)
migration
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

1 Answer

Erick Ramirez avatar image
Erick Ramirez answered Erick Ramirez commented

I recommend you use DataStax Bulk Loader (DSBulk).

With DSBulk, you can load and unload data from a table efficiently in CSV or JSON format. It is 4x faster than the CQL COPY command and doesn't suffer from the same limitations as COPY.

Here are the high-level steps:

  1. Run DSBulk's unload command to export the data from the old table.
  2. Transform the exported data so the columns match the schema of the new table.
  3. Run DSBulk's load command to bulk-load the data to the new table.

Have a look at the reference section of the docs for unloading data and loading data examples. If you're interested, DSBulk also has a count command you could use for verification when you have completed your migration.

Here are some blog posts with detailed examples from Brian Hess you'll find useful:

Cheers!

2 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Sun_P avatar image Sun_P commented ·
Thank you for your answer.
0 Likes 0 ·
Erick Ramirez avatar image Erick Ramirez ♦♦ Sun_P commented ·
Happy to help. Cheers!
0 Likes 0 ·