Feature/external deployment replication

[nouseforaname]

Hi there, I'd like to be able to do replica setups with the release. Here's my idea, let me know what you think of the approach.

Feature or Bug Description

it adds a jobs and a few config options to enable cross deployment replication for the pxc release.

Motivation

I want continuous backups of my running pxc cluster

Related Issue

If this PR was first opened as an issue, please provide the link to that issue here.

[abg]

Thanks! I will try to take a look a this and provide some feedback by Monday.

[abg]

I reviewed this and left a trail of comments.

Beyond the concerns already noted, I would like to see some answer for Galera <-> Galera replication - even if that's simply "replica must only be a single node for now". I.e. I think it would be fine for a template in the pxc-replicator job to abort if the numbers of MySQL nodes != 1. Robustly handling cluster to cluster async replication could be interesting too, but would be much more work.

I would also encourage pxc-replicator to be written in a Go module that is more robust and easier to test and maintain than the current bash scripting.

[abg]

Just a note: RESET MASTER is deprecated in v8.4 and removed in MySQL v9, replaced with RESET BINARY LOGS AND GTIDS.

I understand this is likely used here since pxc-release still supports MySQL v8.0 which still requires this syntax.

[nouseforaname]

yeah, what's the preferred way of handling this?

I noticed while reading docs that they're still cleaning up the weirdo naming..
I tried already using the replica/source naming as far as possible but didn't know if it would be idempotent :/

https://dev.mysql.com/doc/refman/8.4/en/replication-howto-repuser.html

it seems that the permissions still does not have an alternative?

[abg]

Yeah, v8.0 (and earlier) only support RESET MASTER and v8.4 only supports RESET BINARY LOGS ...

We've typically dealt with this kind of thing with version checks:

if mysql_major_minor() == "8.0":  # <- Delete branch when we don't care about old version anymore
   do_legacy_sql_stuff() 
else:
   do_latest_sql_stuff()
end

That could potentially be done at template rendering time by looking at the p('mysql_version') property.

MySQL v8.0 is EOL (although we'll probably get one more patch release from Percona for v8.0.46), so there is an argument to only support v8.4 and onwards. Maybe guard against using this feature on v8.0?

# In some pxc-replicator template
<%- if p('mysql_version') == "8.0" -%>
raise "Unsupported <helpful error message>"
<%- end -%>

[nouseforaname]

hmm. How'd that play with cf-depl?

It seems that 8.4 is still experimental cloudfoundry/cf-deployment@3870c60

I'm not sure if there are any reasons why it's still setup to default to 8.0 outside of being cautious.

I'm going to ask around what the plans for runtime are in regards to switching the default. For now I guess I'll try to find all relevant places and sprinkle some ifs

[abg]

My thinking here was that it could be reasonable to not support this replication feature for EOL MySQL versions to simplify things.

However this exposes that, as pxc-release maintainers, we have been bad netizens of cf-d and have not updated the default MySQL version to v8.4. I'll try to follow-up on that.

I think it's okay to add a version check to support both the old and new syntax for now.

[abg]

typo: laset => last

[nouseforaname]

🧹

[abg]

This seems a little brittle. I would probably actually query the replica and validate the IO_Thread and SQL_Thread are in a healthy state.

[nouseforaname]

first of all, sorry for the horrible PR ( I'm going to excuse myself by having had to learn a lot about pxc and mysql) and second thank you for the in depth review.

I marked this as draft for now. I set up ci to deploy the current state of the release and iterate on it until this is ready.

As far as: should this be go code? Yes :). I think I misunderstood you when I asked in slack about if this should be added to galera-agent and I read you answer as don't add go code totally on me. Having said that, I'll need some time to get this done.

There's one thing that I already can answer, my use case is definitely running an external single instance replica to offload backups from a cf-deployment cluster. I'd like to avoid trying to create the everything PR but I'm happy to iteratively improve until we get there?

My suggestion would be this, I'd like to keep the bash for now, until the general workflow of steps taken in the configuration process is reasonable. once we're there, I'll create a go implementation instead of the bash madness?

Again, thank you for the feedback.

[abg]

I'd like to avoid trying to create the everything PR but I'm happy to iteratively improve until we get there?

Yep, apologies for any confusing feedback. Some of my comments were "nice-to-have-for-the-future" and I didn't make that super obvious. I wouldn't expect a perfect implementation in this initial PR.

Targeting just cf-deployment use cases, mysqldump is probably fine and a single replica is fine.

I think requiring TLS for the MySQL replication channel (and mysqldump / provisioning channel) would be ideal - and at least TLS being a supported configuration. However, mutual TLS and other advanced use cases are not necessary.

My suggestion would be this, I'd like to keep the bash for now, until the general workflow of steps taken in the configuration process is reasonable. once we're there, I'll create a go implementation instead of the bash madness?

Yep, I think that's fine for now. A Go implementation doesn't need to be part of this PR, but I think it's a good end state for testability and alignment with other tooling in the release.

[nouseforaname]

@abg I think I addressed all of the comments, let me know if the current setup is any better.

[abg]

Apologies for the slow feedback. This is improved, but not deployable yet.

I've left some more feedback.

[abg]

I didn't think deeply about this during my initial review pass so up-front apologies for not surfacing this earlier.

These certificates are going to be read by the mysqld process running in the pxc-mysql bpm context. That job currently would not have access to the pxc-replicator job certificates without adjusting the bpm.yml.

Rather than doing this, I think maybe just configure certificates on the pxc-mysql job. This should be the point of the pxc-mysql tls.client.{certificate,key} properties. If configured then you can just set SOURCE_SSL_{CERT,KEY} = /var/vcap/jobs/pxc-mysql/certificates/client-{cert,key}.pem

[abg]

SOURCE_SSL_{CERT,KEY} is only needed if you want x509 authentication (mTLS).

Not necessarily bad a security perspective but the replication user would need to be specifically configured for it.

i.e.

CREATE USER $replicationUSER REQUIRE X509  .... ;
-- or
CREATE USER $replicationUSER REQUIRE SUBJECT '/CN=some-client-identity' ...;

See: MySQL CREATE USER docs

As I mentioned in an earlier comment, mutual TLS is not a requirement for an MVP but maybe a nice-to-have. If you need it, it would add it as an option to user creation.

[abg]

My thinking here was that it could be reasonable to not support this replication feature for EOL MySQL versions to simplify things.

However this exposes that, as pxc-release maintainers, we have been bad netizens of cf-d and have not updated the default MySQL version to v8.4. I'll try to follow-up on that.

I think it's okay to add a version check to support both the old and new syntax for now.

[abg]

Ideally we would apply the principal of least privilege here. Otherwise this will get flagged by security scans.

Replication doesn't need full SELECT access on tables from the primary - it just needs the privilege to stream binary logs.

Similarly the backup user just needs minimal privileges for a backup, but not necessarily binlog streaming privileges.

[abg]

Worth noting that best practice would also set SOURCE_SSL_VERIFY_SERVER_CERT (hostname verification).

This would require the primary's TLS certificate to contain the bosh dns server name (provided by the link here) Bosh docs.

In the context of cf-d this is usually sql-db.service.cf.internal but would require explicit configuration.

[abg]

Any reason not to set these flags via the my.cnf? (i.e. source{,.admin}.mysql.cnf)

[abg]

Could we reuse the mysql link and just expose more properties on it? It already has port and mysql_version

If we are pulling all this from the "primary" deployments link anyway, I was imagining something like:

# primary deployment manifest
- instance_groups:
   - name: mysql
      jobs:
      - name: pxc-mysql
         ...
         provides: mysql: {as: mysql, shared: true}


# replica deployment manifest
- instance_groups:
   - name: mysql
      jobs:
      - name: pxc-replicator
         ...
         consumes: source: {from: mysql, deployment: ((primary_deployment_name)) }

Plus or minus some bosh syntax cleanup here.

Maybe a second link isn't so bad, but it adds some more maintenance overhead where if we expose another mysql job property we have to maintain it in two places going forward.

[nouseforaname]

My reasoning was mostly about being specific what the link type is used for and not wanting to introduce additional properties to existing consumers to avoid confusion. I have no strong feelings about either direction.

[abg]

I don't think these are used anymore and should be cleaned up.

[abg]

With bpm this is probably not super useful - the pid gets dropped as soon as the job starts and monit will treat the job as healthy.

But this made me wonder if we even really need monit for this use case. See my comments on the monit template.

[abg]

Maybe it makes sense to just allow allowed_remote_backup_hosts to be configurable in some way. This seems similar to remote_admin_access and doesn't necessarily need to be tied to replication.