Fix GIS grammars for cross-version compatibility and update documentation by vinperothas · Pull Request #425 · Percona-QA/percona-qa

vinperothas · 2026-03-26T06:49:08Z

Fix GIS grammars for cross-version compatibility and update documentation

- Update gis.yy and linestring.yy to use ST_ prefix for all spatial
  functions, ensuring compatibility across MySQL versions.

- Standardize CAST(... AS SIGNED) across all GIS grammars to resolve
  syntax errors in newer MySQL parsers.

- Explicitly set SRID 0 in geometry constructors (ST_GeomFromText, etc.)
   to allow the optimizer to utilize spatial indexes in 8.0+.

- Improve README.md with detailed setup and execution steps for:
    * gis_functional.yy (via wkt2sql.pl and --post-gendata-sql)
    * gis.yy (self-contained stress testing)
    * linestring.yy (via osm_to_sql.py for real-world topology)

https://perconadev.atlassian.net/browse/PS-10475

…tion - Update gis.yy and linestring.yy to use ST_ prefix for all spatial functions, ensuring compatibility across MySQL versions. - Standardize CAST(... AS SIGNED) across all GIS grammars to resolve syntax errors in newer MySQL parsers. - Explicitly set SRID 0 in geometry constructors (ST_GeomFromText, etc.) to allow the optimizer to utilize spatial indexes in 8.0+. - Improve README.md with detailed setup and execution steps for: * gis_functional.yy (via wkt2sql.pl and --post-gendata-sql) * gis.yy (self-contained stress testing) * linestring.yy (via osm_to_sql.py for real-world topology)

mchawla16 · 2026-05-22T12:39:02Z

Thanks Vin for the changes.

Few points:

Should we create a gendata zz file instead of using post-gendata-sql since for gis the default data generated is not very useful?
Does the linestring.sql have sufficient data, or do we need more data in it?

saikumar-vs

Thanks Vin for working on adding support for latest linestring.sql file to usage/readme and ensuring compatibility with 5.7 and 8.0. Couple of comments to take care, it dosent block the PR but good to have stuff.

saikumar-vs · 2026-06-10T05:53:28Z

+    parser.add_argument("city", help="Name of the city (e.g., 'San Francisco')")
+    parser.add_argument("output", help="Output SQL file")
+    parser.add_argument("--table", default="linestring", help="Table name")
+    parser.add_argument("--mysql-version", choices=["5.7", "8.0"], default="8.0")


Please check and add 8.4 as well

saikumar-vs · 2026-06-16T06:55:13Z

+	  --vardir=$VARDIR \
+	  --post-gendata-sql=conf/gis/linestring.sql \
+	  --grammar=conf/gis/linestring.yy \
+	  --threads=2 \


same as below

saikumar-vs · 2026-06-16T06:55:19Z

+  	  --basedir=$BASEDIR \
+  	  --vardir=$VARDIR \
+  	  --grammar=conf/gis/gis.yy \
+  	  --threads=2 \


id recommend to keep the example with --threads=1, since we dont have good mutithreaded analysis (automatic yet), so keeping in 1 can be still debugged fairy easily in the case of a crash.

Copilot

Pull request overview

This PR updates the RQG GIS grammars and supporting assets to improve cross-MySQL-version compatibility (ST_ function usage, SIGNED casts) and adds documentation + helper tooling to run GIS-focused workloads.

Changes:

Updated GIS grammars to prefer ST_-prefixed functions and standardized numeric casts to CAST(... AS SIGNED).
Added runnable GIS setup artifacts (schema/data samples) plus an OSM-to-SQL generator script.
Expanded randgen/conf/gis/README with execution steps for functional, linestring (OSM-driven), and stress grammars.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
randgen/conf/gis/schema.sql	Adds a minimal GIS schema + seed rows for quick functional runs.
randgen/conf/gis/sample_wkt.txt	Adds sample WKT inputs for quick testing.
randgen/conf/gis/README	Reworks documentation with setup/execution instructions for the GIS grammars and tooling.
randgen/conf/gis/osm_to_sql.py	Adds a script to fetch OSM data via Overpass and generate MySQL spatial inserts.
randgen/conf/gis/linestring.yy	Updates the linestring grammar to use `ST_`-prefixed functions and SRID 0 for constructed polygons/lines.
randgen/conf/gis/linestring.sql	Adds a sample dataset/DDL for the linestring grammar.
randgen/conf/gis/gis.yy	Updates the main GIS stress grammar to use `ST_`-prefixed functions and standardized numeric casts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+INSERT INTO t_geom (id, g) VALUES
+  (1, ST_GeomFromText('POINT(0 0)')),
+  (2, ST_GeomFromText('POINT(10 10)')),
+  (3, ST_GeomFromText('LINESTRING(0 0, 10 10)'));


+  ST_Intersects(g, ST_GeomFromText($wkt)) |
+  ST_Contains(ST_GeomFromText($wkt), g) |
+  ST_Distance(g, ST_PointFromText($point)) < 5 |


+	ST_POINTFROMTEXT(' point_wkt ') |
+	ST_POINTFROMTEXT(' point_wkt ') |
+	ST_POINTFROMTEXT(' point_wkt ') |


+	ST_MPOINTFROMTEXT(' multipoint_wkt ') ;

 linestring:
-	/*executor1 LINESTRINGFROMTEXT(' */ /*executor2 ST_LINEFROMTEXT(' */ linestring_wkt ') |
-	/*executor1 LINESTRINGFROMTEXT(' */ /*executor2 ST_LINEFROMTEXT(' */ linestring_wkt ') |
-	/*executor1 EXTERIORRING( */ /*executor2 ST_EXTERIORRING( */ polygon ) |
-	/*executor1 INTERIORRINGN( */ /*executor2 ST_INTERIORRINGN( */ polygon , returns_integer ) ;
+	ST_LINEFROMTEXT(' linestring_wkt ') |
+	ST_LINEFROMTEXT(' linestring_wkt ') |


 	geometry_1d | geometry_2d |
-	/*executor1 GEOMETRYFROMTEXT(' */ /*executor2 ST_GEOMETRYFROMTEXT(' */ geometry_wkt ') |
-	/*executor1 GEOMETRYN( */ /*executor2 ST_GEOMETRYN( */ geometry_collection , returns_integer ) |
+	ST_GEOMETRYFROMTEXT(' geometry_wkt ') |


@@ -0,0 +1,3 @@
+DROP TABLE IF EXISTS linestring;
+CREATE TABLE linestring (pk INTEGER NOT NULL PRIMARY KEY, linestring_key GEOMETRY NOT NULL, linestring_nokey GEOMETRY NOT NULL, SPATIAL INDEX(linestring_key), SPATIAL INDEX(linestring_nokey)) ENGINE=InnoDB;
+INSERT INTO linestring (pk, linestring_key, linestring_nokey) VALUES (1, GeomFromText('LINESTRING(77.5369751 8.105712, 77.5369103 8.1059789, 77.5369228 8.1059976, 77.53695 8.106012, 77.5371559 8.1060579)'), GeomFromText('LINESTRING(77.5369751 8.105712, 77.5369103 8.1059789, 77.5369228 8.1059976, 77.53695 8.106012, 77.5371559 8.1060579)'));


+def generate_insert_statement(pk: int, wkt: str, table_name: str, use_st_prefix: bool) -> str:
+    func_name = "ST_GeomFromText" if use_st_prefix else "GeomFromText"
+    return (
+        f"INSERT INTO {table_name} (pk, linestring_key, linestring_nokey) "
+        f"VALUES ({pk}, {func_name}('{wkt}'), {func_name}('{wkt}'));"
+    )


+    print(f"Querying Overpass API for city: {city_name}...", file=sys.stderr)
+
+    # Send as POST data, not as URL parameters
+    response = requests.post(overpass_url, data={'data': query}, headers=headers)


+	  --basedir=$BASEDIR \
+	  --vardir=$VARDIR \
+	  --post-gendata-sql=conf/gis/linestring.sql \
+	  --grammar=conf/gis/linestring.yy \
+	  --threads=2 \


+	Use the osm_to_sql.py script to fetch data from OpenStreetMap for any city.
+	  python3 osm_to_sql.py "Kanyakumari" linestring.sql --mysql-version 5.7
+


vinperothas requested review from mchawla16 and saikumar-vs March 26, 2026 06:50

vinperothas force-pushed the gis branch from ed6949b to 9bf0f5a Compare March 26, 2026 14:16

vinperothas force-pushed the gis branch from 9bf0f5a to b00ac08 Compare April 30, 2026 07:21

vinperothas changed the title ~~Add a sample GIS testing grammar and runner~~ Fix GIS grammars for cross-version compatibility and update documentation Apr 30, 2026

vinperothas requested a review from Grishma123-Eng May 4, 2026 04:28

mohitj1988 force-pushed the master branch from 548e03d to b5053df Compare May 22, 2026 08:11

mohitj1988 force-pushed the master branch from c38fb51 to c866054 Compare June 4, 2026 13:52

saikumar-vs reviewed Jun 16, 2026

View reviewed changes

vinperothas requested a review from Copilot June 25, 2026 12:58

Copilot started reviewing on behalf of vinperothas June 25, 2026 12:58 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

mohitj1988 force-pushed the master branch 2 times, most recently from 5b2863f to b523a13 Compare June 30, 2026 19:15

Navigation Menu

Uh oh!

Fix GIS grammars for cross-version compatibility and update documentation - #425

Fix GIS grammars for cross-version compatibility and update documentation#425
vinperothas wants to merge 1 commit into
masterfrom
gis

vinperothas commented Mar 26, 2026 •

edited

Loading

Uh oh!

mchawla16 commented May 22, 2026

Uh oh!

saikumar-vs left a comment •

edited

Loading

Uh oh!

saikumar-vs Jun 10, 2026

Uh oh!

saikumar-vs Jun 16, 2026

Uh oh!

saikumar-vs Jun 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		Use the osm_to_sql.py script to fetch data from OpenStreetMap for any city.
		python3 osm_to_sql.py "Kanyakumari" linestring.sql --mysql-version 5.7

Uh oh!

Conversation

vinperothas commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mchawla16 commented May 22, 2026

Uh oh!

saikumar-vs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saikumar-vs Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

saikumar-vs Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

saikumar-vs Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vinperothas commented Mar 26, 2026 •

edited

Loading

saikumar-vs left a comment •

edited

Loading

saikumar-vs Jun 16, 2026 •

edited

Loading