Merge pull request #102 from ParclLabs/zach/dat-121-limit-total-cap-truncation-warning

fix(DAT-121): make property_v2 `limit` a total cap and warn on silent truncation

Author: zshamroukh

CHANGELOG.md

@@ -1,3 +1,13 @@
+### v1.18.0
+- **`property_v2.search.retrieve`: `limit` is now a cap on the total number of properties returned, not a page size.** Pagination is handled internally to satisfy it. Previously, passing *any* explicit `limit` silently disabled auto-pagination, so `limit=1000` returned one page of 1,000 and discarded every remaining match with no error or warning. Calls with `limit <= 50000` are unaffected — same request, same results.
+- **`limit` above 50,000 now paginates instead of failing.** Previously the request was rejected by the API with `422 limit input should be less than or equal to 50000`.
+- **Partial results now warn instead of passing silently.** When `limit` withholds matching data, a `ParclLabsTruncationWarning` reports how many properties were returned versus how many matched (emitted once per session). Note credits are charged per *property* returned, not per event, and because the returned DataFrame is event-level, `len(df)` is not bounded by `limit`.
+- **Failed pages during pagination are now retried and reported.** Pages are retried up to 3 times with exponential backoff; if any still fail the result is returned with a `ParclLabsIncompleteResultWarning` and the failed offsets are listed in `metadata["incomplete_pages"]`. Previously a failed page was printed and skipped, returning short data indistinguishable from complete data.
+- Added a pagination integrity check that warns if the assembled pages do not yield the expected number of distinct properties.
+- New warning categories in `parcllabs.warnings` (`ParclLabsWarning`, `ParclLabsTruncationWarning`, `ParclLabsIncompleteResultWarning`) so callers can silence or escalate these via standard `warnings` filters.
+- Fixed an internal `auto_paginate` flag leaking into the request query string.
+- Fixed `_get_metadata` mutating the caller's raw first-page response via a shallow copy.
+
 ### v1.17.2
 - Added configurable request timeout to `ParclLabsClient`. Defaults to 10s connect / 90s read. Customizable via the `timeout` parameter on client instantiation.
 

README.md

@@ -480,6 +480,46 @@ Gets a list of unique properties and their associated metadata and events based
 
 **NOTE:** Use the `limit` parameter to specify the number of matched properties to return. If `limit` is not provided, all matched properties will be returned. Conceptually, you should set the `limit` to retrieve a sample of properties, and then if you want to retrieve all properties, make the same request again without the `limit` parameter.
 
+`limit` is a cap on the total number of **properties** returned, and pagination is handled for you — values larger than the API's 50,000 per-request maximum are fetched across multiple pages rather than rejected. Two things to keep in mind:
+
+- **Credits are charged per property returned, not per event.**
+- The returned DataFrame is event-level, so `len(df)` is *not* bounded by `limit` — a single property can contribute many rows.
+
+If `limit` caps the result below the number of matching properties, a `ParclLabsTruncationWarning` is emitted (once per session) and both counts are available in the returned metadata. If any page fails after retries, the data is still returned but a `ParclLabsIncompleteResultWarning` is raised and the failed offsets are listed in `metadata["incomplete_pages"]`.
+
+Both conditions are worth checking programmatically, especially in a loop over many markets where a warning is easy to miss:
+
+```python
+results, metadata = client.property_v2.search.retrieve(parcl_ids=[2900187], limit=5)
+
+counts = metadata["results"]
+if counts["returned_count"] < counts["total_available"]:
+    print(
+        f"Truncated: got {counts['returned_count']:,} of "
+        f"{counts['total_available']:,} properties. Raise `limit`, or omit it entirely."
+    )
+
+if metadata.get("incomplete_pages"):
+    print(f"Incomplete: pages failed at offsets {metadata['incomplete_pages']}")
+```
+
+If a short result should be fatal for your pipeline, make those checks `assert`s or raise your own exception — treat a non-empty `incomplete_pages` as an incomplete dataset either way. Both warning types live in `parcllabs.warnings` and can be silenced or escalated with standard `warnings` filters:
+
+```python
+import warnings
+
+from parcllabs.warnings import ParclLabsIncompleteResultWarning, ParclLabsTruncationWarning
+
+with warnings.catch_warnings():
+    # Intentionally sampling? Silence the truncation notice.
+    warnings.filterwarnings("ignore", category=ParclLabsTruncationWarning)
+
+    # Never accept a partial page silently -- make it raise instead.
+    warnings.filterwarnings("error", category=ParclLabsIncompleteResultWarning)
+
+    results, metadata = client.property_v2.search.retrieve(parcl_ids=[2900187], limit=5)
+```
+
 
 Example request, note that only one of `parcl_ids`, `parcl_property_ids`, or `geo_coordinates` can be provided per request:
 

parcllabs/__version__.py

@@ -1 +1 @@
-VERSION = "1.17.2"
+VERSION = "1.18.0"

parcllabs/schemas/schemas.py

@@ -142,11 +142,21 @@ class PropertyV2RetrieveParams(BaseModel):
     )
 
     # Pagination
+    #
+    # No upper bound: `limit` is a cap on the total number of properties returned,
+    # and values above the API's per-request ceiling
+    # (RequestLimits.PROPERTY_V2_MAX) are satisfied by paginating rather than
+    # rejected. Omit to retrieve every matching property.
     limit: int | None = Field(
         default=None,
         ge=1,
-        le=RequestLimits.PROPERTY_V2_MAX.value,
-        description=f"Number of results to return (max: {RequestLimits.PROPERTY_V2_MAX.value})",
+        description=(
+            "Maximum number of properties to return in total. Values above "
+            f"{RequestLimits.PROPERTY_V2_MAX.value} are fetched across multiple pages. "
+            "Omit to retrieve all matching properties. Credits are charged per property "
+            "returned; the returned DataFrame is event-level, so len(df) is not bounded "
+            "by this value."
+        ),
     )
 
     # Additional parameters