question

Ryan Quey avatar image
Ryan Quey asked dmitri.bourlatchkov_151846 commented

Is there support for partitioning vertices in DSE Graph 6.7+ to handle supernodes?

I'm trying to manage supernodes in DSE Graph, and came across this blog post by Expero. It mentions support for partitioning/cutting vertexes in DSE Graph so that one supernode can be divided among two or more partitions (I think the blog was written based on DSE v. 5.1 or 5.0).

However, as of 6.0 it appears that "Partitioned vertex tables (PVT) are removed. (DSP-13676)" (according to the release notes).

It does look like the command to partition based on an edge is given for 6.0 docs:

schema.vertexLabel('author').partition().inE('created').add()

But then in the 6.7 docs I don't see support for the `partition` step.

Is there still support for cutting vertexes in 6.7 and/or 6.8? If so, how is it implemented and what is its behavior?

graph
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

2 Answers

dmitri.bourlatchkov_151846 avatar image
dmitri.bourlatchkov_151846 answered dmitri.bourlatchkov_151846 commented

In DSE 6.8 Core Graph supports custom partitioning for vertices and edges by means of `.partitionBy` and `.clusterBy` methods in the Schema API.


https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/graph/reference/schema/refEdgeLabel.html

5 comments Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey commented ·

Ok great. So is it correct to say that in 6.8 Core Graph, by default, edges will be stored in same partition as the adjacent vertices, but you can specify partition key using `.partitionBy` in order to store the edge in a different partition?

Also, it doesn't look like the same feature is available in 6.7, since you can't set partition key for edges in 6.7, is that correct?

0 Likes 0 ·
dmitri.bourlatchkov_151846 avatar image dmitri.bourlatchkov_151846 Ryan Quey commented ·

By default edge partitions are based on the full primary key of the incident vertex on "incoming" edge side. This may or may not be equivalent to the partition key of the vertex.

Re: DSE 6.7 - it does not support this as it only has Classic Graph. This is about Core Graph, which is available in DSE 6.8+

1 Like 1 ·
Ryan Quey avatar image Ryan Quey dmitri.bourlatchkov_151846 commented ·

Just to follow up: I tested different PKs in 6.8, and found that by default edge partitions are based on the partition key of the origin vertex. Reading your comment again, I realize that the origin vertex is probably what you meant by "incoming" edge side, but just wanted to write this to clarify this for future readers.

But I did find that the edge partition key was by default using the partition key of the origin vertex, not the full primary key. E.g., for vertex and edge labels created in Gremlin like this:

schema.vertexLabel('person').ifNotExists().
    partitionBy('partition_key', Text).
    clusterBy('uuid', Text).
    create()
schema.edgeLabel('likes').ifNotExists().from('person').to('person').create()

The edges end up with this PK:

PRIMARY KEY (out_partition_key, out_uuid, in_partition_key, in_uuid)
1 Like 1 ·
Show more comments
Show more comments
Erick Ramirez avatar image
Erick Ramirez answered Ryan Quey commented

I'm not fully across it but I have a feeling that we never got around to implementing a solution in DSE 6.7 or 6.8.

I'm going to reach out internally and get the Graph devs to respond to your question directly.

As a side note, we've had an extended 4-day long weekend so there will be delays in responses. Cheers!

1 comment Share
10 |1000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Ryan Quey avatar image Ryan Quey commented ·

Great, thanks for keeping me in the loop

0 Likes 0 ·