Tagging and Untagging servers can lead to an unexpected state

[pri1712]

Background:
I tagged a few servers with a tenant ID (Cell19) but after a while I realized that I had incorrectly tagged some servers as the actual AZ distribution did not match with the expected AZ distribution. So i untagged a few servers but I retagged other servers to have the right AZ distribution.

Issue:
This is leading to a state wherein the Helix/Zk thinks that the older servers are still part of the tenant but the server does not have the old tenants tag on it.
The new servers have the tenants tag but theyre not actually ingesting anything.

List of instances on the tenant page

Server that was added later on( this is not doing any work)

Server that was removed from the tenant. As you can see, it does not have the tenant tag (tenant was 'Cell19')

[Jackie-Jiang]

Is this an upsert table?
You need to rebalance the table to re-position the segments to the correct servers. Pinot might not move segments without rebalance to protect upsert state (keep the entire upstream partition on the same server)

[pri1712]

no this was a vanilla realtime table

[Jackie-Jiang]

In that case, the new consuming segment should be assigned to the new servers. You'll need to run a rebalance to re-assign the existing segments.

[pri1712]

yeah that assignment of newer segments to new servers didnt happen. fyi this is a pretty big cluster with 100s of TBs of data, could batched updates of ZK and control plane be causing this or some slowness in the control path. because this is clearly an issue with consensus

[Jackie-Jiang]

@pri1712 Do you have explicit instance assignment? If so, you'll need to re-assign them.
Do you want to move existing segments, or just move the new segments?

[pri1712]

didnt want to move existing segments, just wanted new segments to be created on the newly assigned instances to the tenant.

[Jackie-Jiang]

Try the controller API to reassign the instances