Skip to content

Expand an unnested record type into a Cascades graph #4626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
g31pranjal merged 6 commits into from
Oct 9, 2026
Merged

Expand an unnested record type into a Cascades graph #4626

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
34dbd53
Expand an unnested record type into a Cascades graph
g31pranjal Sep 28, 2026
a12245d
address comments
g31pranjal Oct 7, 2026
214b59c
nit
g31pranjal Oct 7, 2026
0a23c13
Drop the zero-based ordinality argument removed by #4704
g31pranjal Oct 8, 2026
32968ff
address comments
g31pranjal Oct 9, 2026
ee97c89
Merge branch 'main' into apple/g31pranjal/unnested/synthetic-type-expand
g31pranjal Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions ...ord-layer-core/src/main/java/com/apple/foundationdb/record/metadata/JoinedRecordType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@
import com.apple.foundationdb.record.provider.foundationdb.FDBStoredRecord;
import com.apple.foundationdb.record.provider.foundationdb.FDBSyntheticRecord;
import com.apple.foundationdb.record.provider.foundationdb.IndexOrphanBehavior;
import com.apple.foundationdb.record.provider.foundationdb.RecordDoesNotExistException;

Check notice on line 33 in fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/JoinedRecordType.java

View workflow job for this annotation

GitHub Actions / coverage 

File coverage: 98.3% (58/59 lines) | Changed lines: 100.0% (1/1 lines)
import com.apple.foundationdb.record.query.plan.cascades.AccessHint;
import com.apple.foundationdb.record.query.plan.cascades.GraphExpansion;
import com.apple.foundationdb.tuple.Tuple;
import com.google.protobuf.Descriptors;
import com.google.protobuf.Message;
Expand Down Expand Up @@ -216,4 +218,11 @@
return typeBuilder.build();
}

@Nonnull
@Override
@API(API.Status.INTERNAL)
public GraphExpansion expand(@Nonnull final AccessHint accessHint) {
throw new UnsupportedOperationException("cannot expand an index defined on a " + getClass().getSimpleName());
}

}
17 changes: 17 additions & 0 deletions fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/RecordType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,13 @@
import com.apple.foundationdb.annotation.API;
import com.apple.foundationdb.record.RecordMetaData;
import com.apple.foundationdb.record.RecordMetaDataProvider;
import com.apple.foundationdb.record.metadata.expressions.KeyExpression;

Check notice on line 26 in fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/RecordType.java

View workflow job for this annotation

GitHub Actions / coverage 

File coverage: 98.1% (51/52 lines) | Changed lines: 100.0% (7/7 lines)
import com.apple.foundationdb.record.query.plan.cascades.AccessHint;
import com.apple.foundationdb.record.query.plan.cascades.ExpansionVisitor;
import com.apple.foundationdb.record.query.plan.cascades.GraphExpansion;
import com.apple.foundationdb.record.query.plan.cascades.Quantifier;
import com.apple.foundationdb.tuple.Tuple;
import com.google.common.collect.ImmutableSet;
import com.google.protobuf.Descriptors;

import javax.annotation.Nonnull;
Expand Down Expand Up @@ -235,6 +240,18 @@
return false;
}

@Nonnull
@API(API.Status.INTERNAL)
public GraphExpansion expand(@Nonnull final AccessHint accessHint) {

@normen662 normen662 Oct 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

final Quantifier.ForEach quantifier =
Quantifier.forEach(ExpansionVisitor.createBaseRef(metaData.getRecordTypes().keySet(),
ImmutableSet.of(getName()), metaData.getPlannerType(getName()), null, accessHint));
return GraphExpansion.builder()
.addQuantifier(quantifier)
.addResultValue(quantifier.getFlowedObjectValue())
.build();
}

@Override
public String toString() {
StringBuilder str = new StringBuilder();
Expand Down
7 changes: 7 additions & 0 deletions ...-layer-core/src/main/java/com/apple/foundationdb/record/metadata/SyntheticRecordType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@
import com.apple.foundationdb.record.metadata.expressions.KeyExpression;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore;
import com.apple.foundationdb.record.provider.foundationdb.FDBSyntheticRecord;
import com.apple.foundationdb.record.provider.foundationdb.IndexOrphanBehavior;

Check notice on line 28 in fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/SyntheticRecordType.java

View workflow job for this annotation

GitHub Actions / coverage 

File coverage: 94.4% (17/18 lines) | Changed lines: N/A (no executable lines)
import com.apple.foundationdb.record.query.plan.cascades.AccessHint;
import com.apple.foundationdb.record.query.plan.cascades.GraphExpansion;
import com.apple.foundationdb.tuple.Tuple;
import com.google.protobuf.Descriptors;

Expand Down Expand Up @@ -102,6 +104,11 @@
@Nonnull
public abstract CompletableFuture<FDBSyntheticRecord> loadByPrimaryKeyAsync(FDBRecordStore store, Tuple primaryKey, IndexOrphanBehavior orphanBehavior);

@Nonnull
@Override
@API(API.Status.INTERNAL)
public abstract GraphExpansion expand(@Nonnull AccessHint accessHint);

@Override
public String toString() {
StringBuilder str = new StringBuilder();
Expand Down
91 changes: 91 additions & 0 deletions ...d-layer-core/src/main/java/com/apple/foundationdb/record/metadata/UnnestedRecordType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,32 @@
import com.apple.foundationdb.record.RecordCoreException;
import com.apple.foundationdb.record.RecordMetaData;
import com.apple.foundationdb.record.RecordMetaDataProto;
import com.apple.foundationdb.record.logging.LogMessageKeys;

Check notice on line 27 in fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/UnnestedRecordType.java

View workflow job for this annotation

GitHub Actions / coverage 

File coverage: 97.3% (107/110 lines) | Changed lines: 100.0% (41/41 lines)
import com.apple.foundationdb.record.metadata.expressions.FieldKeyExpression;
import com.apple.foundationdb.record.metadata.expressions.KeyExpression;
import com.apple.foundationdb.record.metadata.expressions.LiteralKeyExpression;
import com.apple.foundationdb.record.metadata.expressions.NestingKeyExpression;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore;
import com.apple.foundationdb.record.provider.foundationdb.FDBStoredRecord;
import com.apple.foundationdb.record.provider.foundationdb.FDBSyntheticRecord;
import com.apple.foundationdb.record.provider.foundationdb.IndexOrphanBehavior;
import com.apple.foundationdb.record.provider.foundationdb.RecordDoesNotExistException;
import com.apple.foundationdb.record.query.plan.cascades.AccessHint;
import com.apple.foundationdb.record.query.plan.cascades.Column;
import com.apple.foundationdb.record.query.plan.cascades.GraphExpansion;
import com.apple.foundationdb.record.query.plan.cascades.NullableArrayTypeUtils;
import com.apple.foundationdb.record.query.plan.cascades.Quantifier;
import com.apple.foundationdb.record.query.plan.cascades.Reference;
import com.apple.foundationdb.record.query.plan.cascades.expressions.ExplodeExpression;
import com.apple.foundationdb.record.query.plan.cascades.typing.Type;
import com.apple.foundationdb.record.query.plan.cascades.values.FieldValue;
import com.apple.foundationdb.record.query.plan.cascades.values.PromoteValue;
import com.apple.foundationdb.record.query.plan.cascades.values.RecordConstructorValue;
import com.apple.foundationdb.record.query.plan.cascades.values.Value;
import com.apple.foundationdb.record.util.ProtoUtils;
import com.apple.foundationdb.tuple.Tuple;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import com.google.protobuf.Descriptors;
import com.google.protobuf.Message;

Expand All @@ -42,8 +59,10 @@
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;


/**
* A {@linkplain SyntheticRecordType synthetic record type} representing an unnesting of some kind of
* repeated, nested structure. Each record of this type should be associated with one stored record
Expand Down Expand Up @@ -357,4 +376,76 @@
}
return builder.build();
}

@Nonnull
@Override
@API(API.Status.INTERNAL)
public GraphExpansion expand(@Nonnull final AccessHint accessHint) {
Comment thread
g31pranjal marked this conversation as resolved.
Resolved
final GraphExpansion parentExpansion = getParentConstituent().getRecordType().expand(accessHint);
final Map<String, Value> elementValuesByConstituent = new HashMap<>();
final GraphExpansion.Builder builder = GraphExpansion.builder().addAllQuantifiers(parentExpansion.getQuantifiers());
final ImmutableList.Builder<Column<? extends Value>> positionColumns = ImmutableList.builder();
for (final NestedConstituent constituent : getConstituents()) {
final Value elementValue;
if (constituent.isParent()) {
elementValue = Iterables.getOnlyElement(parentExpansion.getResultColumns()).getValue();
} else {
final Value ownerElementValue =
Objects.requireNonNull(elementValuesByConstituent.get(constituent.getParentName()));
final Quantifier.ForEach constituentQuantifier =
constituentQuantifier(ownerElementValue, constituent.getNestingExpression());
builder.addQuantifier(constituentQuantifier);
final Value flowedValue = constituentQuantifier.getFlowedObjectValue();
elementValue = FieldValue.ofOrdinalNumber(flowedValue, 0);
final Value positionValue = PromoteValue.inject(FieldValue.ofOrdinalNumber(flowedValue, 1),
Type.primitiveType(Type.TypeCode.LONG, false));
positionColumns.add(Column.of(Optional.of(constituent.getName()), positionValue));
}
elementValuesByConstituent.put(constituent.getName(), elementValue);
builder.addResultColumn(Column.of(Optional.of(constituent.getName()), elementValue));
}
builder.addResultColumn(Column.of(Optional.of(POSITIONS_FIELD),
RecordConstructorValue.ofColumns(positionColumns.build())));
return builder.build();
Comment thread
normen662 marked this conversation as resolved.
Resolved
}

/**
* Builds the quantifier standing for one constituent: a select over an explode of the constituent's array.
*
* <p>The explode is created {@code WITH ORDINALITY}, so it flows an anonymous {@code (element, ordinal)} struct.
* The ordinal is needed to reconstruct the {@code __positions} field of a synthetic record, without which the
* synthetic primary key cannot be expressed. The struct is asked for as a record constructor, so that the
* element and the ordinal are sub-values a query's values can be matched against.
*
* @param ownerElementValue the record the array hangs off
* @param nestingExpression the constituent's nesting expression
* @return a quantifier flowing {@code (element, ordinal)} structs for the constituent's array
*/
@Nonnull
private static Quantifier.ForEach constituentQuantifier(@Nonnull final Value ownerElementValue,
@Nonnull final KeyExpression nestingExpression) {
final Quantifier.ForEach explodeQuantifier =
Quantifier.forEach(Reference.initialOf(new ExplodeExpression(
FieldValue.ofFieldNames(ownerElementValue, arrayFieldPath(nestingExpression)), true, true)));
return Quantifier.forEach(Reference.initialOf(GraphExpansion.ofQuantifier(explodeQuantifier)
.seal()
.buildSimpleSelectOverQuantifier(explodeQuantifier)));
}

@Nonnull
private static List<String> arrayFieldPath(@Nonnull final KeyExpression nestingExpression) {

@normen662 normen662 Oct 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check if you like this better in MessageHelpers?

@g31pranjal g31pranjal Oct 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't go with MessageHelpers since it doesn't have to do anything with Message. What about in KeyExpression or NullableArrayTypeUtils? I am not fully convinced though

@normen662 normen662 Oct 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyExpression is probably better if this were to be reused.

@normen662 normen662 Oct 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or leave it here. your call

final ImmutableList.Builder<String> pathBuilder = ImmutableList.builder();
KeyExpression current = nestingExpression;
while (current instanceof final NestingKeyExpression nesting) {
final FieldKeyExpression parent = nesting.getParent();
pathBuilder.add(ProtoUtils.toUserIdentifier(parent.getFieldName()));
final var wrapperFanType = NullableArrayTypeUtils.matchArrayWrapper(nesting);
if (wrapperFanType.isPresent()) {
return pathBuilder.build();
}
current = nesting.getChild();
}
pathBuilder.add(ProtoUtils.toUserIdentifier(((FieldKeyExpression)current).getFieldName()));
return pathBuilder.build();
}
}
Loading
Loading