diff --git a/conda-recipe/meta.yaml b/conda-recipe/meta.yaml
index 7c2cd95..71a1532 100644
--- a/conda-recipe/meta.yaml
+++ b/conda-recipe/meta.yaml
@@ -16,7 +16,7 @@ requirements:
build:
- git
- python
- - setuptools>=68.0
+ - setuptools>=68.0, <81.0
- setuptools_scm[toml]>=8.0.0
- cython==3.0.* # [py<314]
- cython>=3.1.3 # [py>=314]
diff --git a/custom_theme/partials/header.html.tpl b/custom_theme/partials/header.html.tpl
index e412c5a..5ede20a 100644
--- a/custom_theme/partials/header.html.tpl
+++ b/custom_theme/partials/header.html.tpl
@@ -126,5 +126,5 @@
}
-
+
diff --git a/docs/getting-started/what_is_pykx.md b/docs/getting-started/what_is_pykx.md
index f2bce8c..543e39b 100644
--- a/docs/getting-started/what_is_pykx.md
+++ b/docs/getting-started/what_is_pykx.md
@@ -14,6 +14,10 @@ _This page briefly describes PyKX, its use cases, and its connection with q/kdb+
**PyKX** is a Python-first interface to **kdb+** (the world's fastest time-series database) and **q** (kdb+'s underlying vector programming language). PyKX is the result of 10+ years of integrations between two languages: Python and q. Its aim is to help users query and analyze huge amounts of in-memory and on-disk time-series data, significantly faster than other libraries.
+- For **Python developers**, PyKX unlocks the speed and power of kdb+ for data processing and storage from within your Python environment. It enables anyone with Python knowledge to apply analytics against vast amounts of data, both in-memory and on-disk, in a fraction of the time, allowing you to focus on getting the best from your data.
+
+- For **q developers**, PyKX brings together Python's data science ecosystem and the power of kdb+'s vector and time-series analytics. This makes them available in both q and Python environments. You can use it to run q code within a Python environment or embed Python analytics within your q session.
+
## Use cases
PyKX supports three main use cases, allowing Python data engineers and data scientists to:
diff --git a/docs/index.md b/docs/index.md
index 4fbec86..651c04b 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,41 +1,67 @@
-# PyKX Documentation
+# PyKX Home
+
+_Welcome to the home of PyKX technical documentation._
+
+[PyKX](./getting-started/what_is_pykx.md) is a Python-first interface to kdb+, the world's fastest time-series database, and its underlying vector programming language q. Whether you're a Python developer looking to harness kdb+'s analytics, or a q developer wanting to leverage Python's data science ecosystem, PyKX bridges both worlds.
+
+## Ready to dive in?
+
+
+
+## What's next?
+
+Discover how PyKX builds upon kdb+ and what's coming soon:
+
+- Submit a [feature request](https://forum.kx.com/c/ideas-feature-requests/10)
+- Stay updated with the latest [release notes](release-notes/changelog.md){target="_blank"}
+- Join our [community channels](https://kx-community.slack.com/) to get assistance
-## About
-
-Welcome to [PyKX](./getting-started/what_is_pykx.md)! PyKX is a Python first interface to the world's fastest time-series database kdb+ and its underlying vector programming language, q.
-
-For Python developers, PyKX unlocks the speed and power of kdb+ for data processing and storage from within your Python environment. It enables anyone with Python knowledge to apply analytics against vast amounts of data, both in-memory and on-disk, in a fraction of the time, allowing you to focus on getting the best from your data.
-
-For q developers, PyKX brings together Python's data science ecosystem and the power of kdb+'s vector and time-series analytics. This makes them available in both q and Python environments. You can use it to run q code within a Python environment or embed Python analytics within your q session.
-
-To begin your journey with PyKX, follow the sections below:
-
-## Documentation overview
-
-!!! home-page "[Get Started](./getting-started/installing.md)"
-
- This section is for new users. Includes everything you need to begin your journey with PyKX: [installation instructions](./getting-started/installing.md) and a [quickstart guide](./getting-started/quickstart.md).
-
-!!! home-page "[Learn](./getting-started/what_is_pykx.md)"
-
- Explore the core concepts of our technology. This section provides you with detailed explanations and access to [KX Academy](https://learninghub.kx.com/courses/introduction-to-pykx/) materials.
-
-!!! home-page "[How To](./user-guide/configuration.md)"
-
- Follow step-by-step guides to take you from initial setup to advanced production systems. You’ll find practical advice and best practices here.
-
-!!! home-page "[Reference](./api/pykx-execution/q.md)"
-
- Find detailed technical references for methods, functions, and APIs. This is your go-to source for all technical definitions and usage examples.
-
-!!! home-page "[Integrations](./user-guide/advanced/numpy.md)"
-
- Learn how to integrate PyKX with other tools and platforms. This section includes guides and compatibility information to ensure seamless interoperability.
-
-!!! home-page "[Releases](./release-notes/changelog.md)"
-
- Stay updated with the latest release notes. You’ll find information on the latest releases/fixes, and previous versions.
-
-!!! home-page "[Help and Support](./help/troubleshooting.md)"
-
- Get assistance with any questions or issues you might have. This section includes [troubleshooting](./help/troubleshooting.md) guides, [FAQs](./help/faq.md), and [support](./help/support.md) contact information.
diff --git a/docs/release-notes/changelog.md b/docs/release-notes/changelog.md
index e2dad40..10f951b 100644
--- a/docs/release-notes/changelog.md
+++ b/docs/release-notes/changelog.md
@@ -4,7 +4,53 @@
The changelog presented here outlines changes to PyKX when operating within a Python environment specifically, if you require changelogs associated with PyKX operating under a q environment see [here](./underq-changelog.md).
-# PyKX 3.1.8
+## PyKX 3.1.9
+
+#### Release Date
+
+2026-04-02
+
+### Additions
+
+- Added `.np()` method to `pykx.serialize` function, this returns the serialized data as a NumPy array.
+- Added support for calling `pykx.deserialize` on NumPy arrays of serialized `K` objects.
+
+ ```python
+ >>> import pykx as kx
+ >>> vec = kx.q('til 10')
+ >>> vec
+ pykx.LongVector(pykx.q('0 1 2 3 4 5 6 7 8 9'))
+ >>> ser = kx.serialize(vec)
+ >>> type(ser)
+
+ >>> ser_np = ser.np()
+ >>> type(ser_np)
+
+ >>> ser_np.dtype
+ dtype('uint8')
+ >>> kx.deserialize(ser_np)
+ pykx.LongVector(pykx.q('0 1 2 3 4 5 6 7 8 9'))
+ ```
+
+- Added `.bin()` method to `pykx.Column` class, to perform a binary search using the data in a `pykx.Column` object.
+
+ ```python
+ >>> import pykx as kx
+ >>> t = kx.q('([] n: 2 4)')
+ >>> t.select(kx.Column('n').bin([1,3,5]))
+ pykx.Table(pykx.q('
+ n
+ -
+ 0
+ 1
+ '))
+ ```
+
+### Fixes and Improvements
+
+- Updated 4.1 to 2026.04.01 for all platforms.
+
+## PyKX 3.1.8
#### Release Date
diff --git a/docs/scripts/strip-q-prompt-copy.js b/docs/scripts/strip-q-prompt-copy.js
new file mode 100644
index 0000000..42e9b9a
--- /dev/null
+++ b/docs/scripts/strip-q-prompt-copy.js
@@ -0,0 +1,150 @@
+// When a user clicks a code copy button, copy the code but remove
+// leading REPL prompts (q), >>>, and Python continuation ...).
+//
+// Add this file to mkdocs extra_javascript as javascript/strip-repl-prompts.js
+// and ensure it loads after the Material copy button script.
+
+(function () {
+
+ // Robust selectors covering Material for MkDocs variations
+ // (theme versions use slightly different class names)
+ const COPY_BUTTON_SELECTORS = [
+ '.md-code .md-clipboard', // older / generic
+ 'button.md-code__copy', // theme internal
+ 'button.md-clipboard', // fallback
+ '.md-code__copy-button' // some custom builds
+ ].join(',');
+
+ // Regex to remove leading REPL prompts (allow optional leading whitespace)
+ // Matches:
+ // q)
+ // >>>
+ // ...
+ const REPL_PROMPT_RE = /^\s*(q\)|>>>|\.\.\.)\s?/;
+
+ // Helper: find nearest ancestor matching a selector
+ function closest(el, selector) {
+ while (el) {
+ if (el.matches && el.matches(selector)) return el;
+ el = el.parentElement;
+ }
+ return null;
+ }
+
+ // Clean the code text by removing REPL prompts at the start of each line
+ function stripReplPrompts(text) {
+ return text
+ .split('\n')
+ .map(line => line.replace(REPL_PROMPT_RE, ''))
+ .join('\n');
+ }
+
+ // Retrieve visible code text from a code block container
+ function getCodeTextFromBlock(container) {
+ if (!container) return '';
+
+ // Typical structure:
+ //
+ //
...
+ //
+ const codeElem =
+ container.querySelector('pre > code') ||
+ container.querySelector('code');
+
+ if (!codeElem) return '';
+
+ // Use textContent to preserve visible text only (no markup)
+ return codeElem.textContent || '';
+ }
+
+ // Clipboard write with modern API and legacy fallback
+ function writeToClipboard(text) {
+ if (navigator.clipboard && navigator.clipboard.writeText) {
+ return navigator.clipboard.writeText(text);
+ }
+
+ // Fallback for older browsers
+ const textarea = document.createElement('textarea');
+ textarea.value = text;
+ textarea.style.position = 'fixed';
+ textarea.style.left = '-9999px';
+ document.body.appendChild(textarea);
+ textarea.select();
+
+ try {
+ document.execCommand('copy');
+ document.body.removeChild(textarea);
+ return Promise.resolve();
+ } catch (err) {
+ document.body.removeChild(textarea);
+ return Promise.reject(err);
+ }
+ }
+
+ // Attach copy interception to all copy buttons
+ function wireCopyButtons() {
+ document.querySelectorAll(COPY_BUTTON_SELECTORS).forEach(btn => {
+
+ // Prevent duplicate binding if MutationObserver re-runs
+ if (btn.__replPromptBound) return;
+ btn.__replPromptBound = true;
+
+ btn.addEventListener('click', function (ev) {
+ try {
+ // Locate the containing code block
+ const codeContainer =
+ closest(btn, '.md-code') ||
+ closest(btn, 'pre') ||
+ closest(btn, 'code') ||
+ btn.parentElement;
+
+ const raw = getCodeTextFromBlock(codeContainer);
+
+ if (!raw) {
+ // If no code found, allow default behavior
+ return;
+ }
+
+ // Remove REPL prompts before copying
+ const cleaned = stripReplPrompts(raw);
+
+ // Override default copy behavior
+ ev.preventDefault();
+ ev.stopPropagation();
+
+ writeToClipboard(cleaned).then(() => {
+ // Optional: emit event for future UI feedback hooks
+ btn.dispatchEvent(
+ new CustomEvent('repl-copy-success', { bubbles: true })
+ );
+ }).catch(() => {
+ // On failure, allow theme fallback behavior
+ });
+
+ } catch (e) {
+ // Fail silently and allow default copy behavior
+ return;
+ }
+
+ }, { capture: false });
+
+ });
+ }
+
+ // Initial binding once DOM is ready
+ document.addEventListener('DOMContentLoaded', () => {
+ wireCopyButtons();
+
+ // Re-bind if new code blocks are dynamically added
+ const observer = new MutationObserver(() => {
+ wireCopyButtons();
+ });
+
+ observer.observe(document.body, {
+ childList: true,
+ subtree: true
+ });
+ });
+
+ })();
+
\ No newline at end of file
diff --git a/docs/stylesheets/2021-edited.css b/docs/stylesheets/2026-edited.css
similarity index 75%
rename from docs/stylesheets/2021-edited.css
rename to docs/stylesheets/2026-edited.css
index 2f7c1ea..cf82b53 100644
--- a/docs/stylesheets/2021-edited.css
+++ b/docs/stylesheets/2026-edited.css
@@ -145,9 +145,9 @@
/* RHS TOC TITLE & NAV TREE TITLE - Euclid Square Bold UPPERCASE 9/12pt */
.md-nav--lifted>.md-nav__list>.md-nav__item--active>.md-nav__link,
.md-nav__title {
- font-weight: 300;
+ font-weight: 600;
letter-spacing: 0.2em;
- opacity: 0.5;
+ opacity: 1;
padding-bottom: .8em;
text-transform: uppercase;
}
@@ -254,3 +254,111 @@
background-color: transparent;
margin-top: 0;
}
+
+ [data-md-color-scheme="kx-light"] .md-header {
+ background-color: #194387;
+ color: white;
+ }
+
+ [data-md-color-scheme="kx-light"] .md-tabs {
+ background-color: #dae6f9;
+ }
+
+ [data-md-color-scheme="kx-light"] .md-tabs__link {
+ color: #2b2f38;
+ }
+
+ [data-md-color-scheme="kx-light"] .md-tabs__link--active {
+ color: black;
+ border-bottom: none;
+ }
+ [data-md-color-scheme="kx-light"] .md-nav--primary .md-nav__title {
+ background-color: #194387;
+ color: white;
+ }
+
+/* Tiles */
+
+.md-typeset div.large-tile-grid {
+ display: grid !important;
+ grid-template-columns: 1fr 1fr !important;
+ gap: 3rem !important;
+}
+
+@media screen and (max-width: 640px) {
+ .md-typeset div.large-tile-grid {
+ grid-template-columns: 1fr;
+ }
+}
+.md-typeset div.large-tile-grid > div {
+ display: flex;
+ flex-direction: column;
+ padding: 1.25rem;
+ border: 1px solid var(--kx-black);
+ border-radius: 10px;
+}
+.md-typeset div.large-tile-grid > div h2 {
+ margin: 0;
+ font-weight: 500;
+}
+.md-typeset div.large-tile-grid > div h2 span {
+ display: block;
+ margin-top: 0.3rem;
+ font-size: 0.9rem;
+ font-weight: normal;
+}
+.md-typeset div.large-tile-grid > div ul {
+ flex-grow: 1;
+ list-style: none;
+ padding: 0;
+ margin: 0;
+}
+.md-typeset div.large-tile-grid > div ul li {
+ margin: 0.75rem 0;
+ font-size: 0.8rem;
+}
+.md-typeset div.large-tile-grid > div a.cta {
+ display: inline-block;
+ align-self: flex-start;
+ margin-top: 0.3rem;
+ padding: 0.5rem 1rem;
+ border-radius: 50px;
+ font-weight: bold;
+ background-color: var(--kx-blue);
+ color: var(--kx-off-white);
+}
+.md-typeset div.large-tile-grid > div a.cta svg {
+ position: relative;
+ top: 3px;
+ display: inline-block;
+ margin-left: 0.3rem;
+ transition: all 200ms ease-in-out;
+ fill: var(--kx-off-white);
+}
+.md-typeset div.large-tile-grid > div a.cta:hover svg {
+ transform: translateX(4px);
+}
+
+.md-typeset div.large-tile-grid > div {
+ display: flex;
+ flex-direction: column;
+ padding: 1.25rem;
+ border: 1px solid #0b0d0e;
+ border-radius: 10px;
+}
+
+[data-md-color-scheme="slate"] .md-typeset div.large-tile-grid > div {
+ border-color: #FFC300;
+ background-color: #0f1114;
+}
+
+[data-md-color-scheme="slate"] {
+ --md-default-bg-color: #181c1f;
+}
+.md-nav__link {
+ user-select: none;
+}
+
+.md-tabs__link {
+ user-select: none;
+}
\ No newline at end of file
diff --git a/docs/stylesheets/mkdocstrings.css b/docs/stylesheets/mkdocstrings.css
index a101a33..3d5a8e9 100644
--- a/docs/stylesheets/mkdocstrings.css
+++ b/docs/stylesheets/mkdocstrings.css
@@ -23,3 +23,23 @@ h2.doc > code {
h3.doc > code {
background: var(--md-code-bg-color);
}
+.md-grid {
+ max-width: 72rem;
+ }
+
+ @media screen and (min-width: 76.25em) {
+ [dir="ltr"] .md-sidebar--primary:not([hidden])~.md-content>.md-content__inner {
+ margin-left: 0.4rem;
+ }
+
+ [dir="ltr"] .md-sidebar--secondary:not([hidden])~.md-content>.md-content__inner {
+ margin-right: 0.4rem;
+ }
+
+ [dir="rtl"] .md-sidebar--primary:not([hidden])~.md-content>.md-content__inner {
+ margin-right: 0.4rem;
+ }
+
+ [dir="rtl"] .md-sidebar--secondary:not([hidden])~.md-content>.md-content__inner {
+ margin-left: 0.4rem;
+ }
\ No newline at end of file
diff --git a/docs/stylesheets/pykx.css b/docs/stylesheets/pykx.css
index a726c37..3bc2ec0 100644
--- a/docs/stylesheets/pykx.css
+++ b/docs/stylesheets/pykx.css
@@ -8,3 +8,4 @@ div.doc-contents:not(.first) {
border-left: .05rem solid var(--md-default-fg-color--lightest);
margin-bottom: 80px;
}
+
diff --git a/docs_archive/kx_for_databricks/clusterSetups.md b/docs_archive/kx_for_databricks/clusterSetups.md
new file mode 100644
index 0000000..6ffdd07
--- /dev/null
+++ b/docs_archive/kx_for_databricks/clusterSetups.md
@@ -0,0 +1,407 @@
+---
+title: KX for Databricks cluster setups
+description: How to use KX for Databricks
+date: September 2024
+author: KX Systems, Inc.,
+tags: PyKX, Databricks, single node, multi node, workflows
+---
+
+# Cluster setups
+
+_The purpose of this page is to help you set up your single node or multi node clusters._
+
+You can run KX for Databricks on both single node and multi-node computes. Choose the compute and RAM requirements depending on your data/pipeline requirements.
+
+For the best experience, read the [KX for Databricks Overview](introduction.md) and the [Get started](getStarted.md) guide first.
+
+Now let's explore examples for two types of cluster setups:
+
+1. [Single node workflow](#1-single-node-workflow)
+2. [Multi node workflow](#2-multi-node-workflow)
+
+!!! info "Pre-requisites"
+
+ The following pre-requisites must be satisfied in order to run the below notebooks:
+
+ - Configuration of single node or multi node compute
+ - Latest version of PyKX must be installed
+ - QLIC env variable must point to a valid kc.lic file
+ - Load Trade, Quote & Execution data from a Databricks Unity Catalogue.
+
+ For more information refer to the [Get Started documentation](getStarted.md).
+
+## 1. Single node workflow
+
+A single node compute runs spark locally and the driver works as both master and worker, with no worker nodes.
+It is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.
+
+Import the required Python libraries:
+
+```python
+import pykx as kx
+import numpy as np
+import pandas as pd
+from datetime import datetime, timedelta
+```
+
+### Hourly OHLC - Pythonic vs. q magic
+
+!!! note "OHLC is the open, high, low, and close price for a given period."
+
+The example below calculates the hourly OHLC in two ways: using regular Python syntax and then the q magic.
+
+!!! example ""
+
+ === "Pythonic"
+
+ Calculate the OHLC using regular Python syntax:
+
+ ```python
+ kx.q("{select Open:first price,High:max price,Low:min price,Close:last price by bucket:0D01 xbar time,sym from x}", trade).head()
+ ```
+
+ This should produce the following output:
+
+ ```
+ bucket sym Open High Low Close
+ 2024.07.04D04:00:00.000000000 AEF 617f 617f 617f 617f
+ BKR 93.21 93.21 93.21 93.21
+ CLMT 2327.99 2327.99 2327.99 2327.99
+ CMG 780f 780f 777.6 780f
+ F.PRA 333.99 336.79 332.6 336.69
+ ```
+
+ === "q magic"
+
+ Calculate the OHLC using q magic:
+
+ ```python
+ %%q
+ select Open:first price,High:max price,Low:min price,Close:last price by bucket:0D01 xbar time,sym from trade
+ ```
+
+ This should produce the following output:
+
+ ```python
+ bucket sym | Open High Low Close
+ ------------------------------------| -------------------------------
+ 2024.07.04D04:00:00.000000000 AEF | 617 617 617 617
+ 2024.07.04D04:00:00.000000000 BKR | 93.21 93.21 93.21 93.21
+ 2024.07.04D04:00:00.000000000 CLMT | 2327.99 2327.99 2327.99 2327.99
+ 2024.07.04D04:00:00.000000000 CMG | 780 780 777.6 780
+ 2024.07.04D04:00:00.000000000 F.PRA | 333.99 336.79 332.6 336.69
+ ```
+
+Notice how both examples produce the same result.
+
+### VWAP
+
+!!! note "VWAP - The average price a security has traded at throughout the day, based on both volume and price."
+
+The example below calculates the volume-weighted average price (VWAP) using q magic:
+
+```python
+%%q
+select vwap:size wavg price by sym from trade
+```
+
+??? example "Output"
+
+ ```
+ sym | vwap
+ ------| --------
+ AEF | 625.727
+ AGND | 228.8684
+ AUY | 251.6101
+ AXJL | 494.2899
+ AXX | 326.6949
+ ```
+
+### Volatility
+
+!!! note "Volatility - A statistical measure of the dispersion of returns for a given security or market index."
+
+The example below calculates volatility for the stock `AEF`.
+
+Define the functions used to calculate the volatility:
+
+```python
+get_volatility_pykx = kx.q("{[bid;ask] {r:0^log[x]-log next x;sqrt ema[0.001] r*r } 0.5*ask+bid}")
+
+def get_volatility(quote_tbl, sym) -> dict:
+ subset_quotes = quote_tbl.loc[
+ (quote_tbl["sym"] == sym)
+ & (quote_tbl["time"] >= timedelta(hours=9, minutes=25))
+ & (quote_tbl["time"] < timedelta(hours=16, minutes=0))
+ ]
+ return {
+ "datetime": subset_quotes["time"],
+ "volatility": get_volatility_pykx(subset_quotes["bid"], subset_quotes["ask"])
+ }
+```
+
+Pass the `quote` table and the `AEF` symbol to the `get_volatility` function:
+
+```python
+get_volatility(quote, 'AEF')
+```
+
+??? example "Output"
+
+ ```python
+ {'datetime': pykx.TimestampVector(pykx.q('2024.07.04D09:25:03.491000000 2024.07.04D09:25:46.166000000 2024.07.04D09:27:..')),
+ 'volatility': pykx.FloatVector(pykx.q('0.003473635 0.003496871 0.003534634 0.003532866 0.003612577 0.003686721 0.003..'))}
+ ```
+
+### Spread
+
+!!! note "Spread - The difference or gap that exists between two prices, rates, or yields."
+
+The example below calculates the moving average over the spread for the stock `AEF`.
+
+Define the functions used to calculate the moving average over the spread:
+
+```python
+def get_mavg_of_spread(bid, ask, mavg_size: int):
+ spreads = ask - bid
+ return np.convolve(spreads, np.ones(mavg_size), mode="same") / mavg_size
+
+kx.q["mavg_of_spread"] = get_mavg_of_spread
+
+def get_spread(quotes: kx.Table, sym: str) -> kx.Table:
+ return kx.q.qsql.select(
+ quotes,
+ columns={"datetime": "time", "spread": "mavg_of_spread[bid; ask; 1000]", "bid":"bid", "ask":"ask"},
+ where=[f'sym=`$"{sym}"', "bid>0", "ask>0", "time>=09:25", "time <16:00"]
+ )
+```
+
+Pass the `quote` table and the `AEF` symbol to the `get_spread` function:
+
+```python
+get_spread(quote,'AEF').head()
+```
+
+??? example "Output"
+
+ ```python
+ datetime spread bid ask
+ 0 2024.07.04D09:25:03.491000000 15.07707 580.67 653.61
+ 1 2024.07.04D09:25:46.166000000 15.07735 600f 630f
+ 2 2024.07.04D09:27:01.015000000 15.07766 588.88 625f
+ 3 2024.07.04D09:27:10.556000000 15.07844 580.67 653.61
+ 4 2024.07.04D09:27:10.557000000 15.07873 580.67 653.61
+ ```
+
+### Slippage
+
+!!! note "Slippage - The difference between the expected price of a trade and the price at which the trade is executed."
+
+The example below calculates the Slippage based on the midpoint between the prevailing quoted bid and ask prices.
+
+Create the `merged` table by performing an as-of-join on the `execs` and `quote` tables:
+
+```python
+merged = kx.q.aj(kx.SymbolVector(["sym", "time"]), execs, quote)
+```
+
+Calculate the Mid,Delta and Slippage columns:
+
+```python
+merged["mid"] = 0.5 * (merged["bid"] + merged["ask"])
+merged["diff"] = np.where(
+ merged["side"] == "BUY", merged["mid"] - merged["price"], merged["price"] - merged["mid"]
+)
+merged["slippage"] = (merged["diff"] / merged["mid"]) * 10000
+
+merged.head()
+```
+
+??? example "Output"
+
+ ```python
+ date sym time price size side venue bid ask bsize asize mode ex mid diff slippage
+ 0 2022.03.31 GE.PRA 2024.07.04D09:30:02.676000000 397.6131 12 BUY venue1 396.51 397.96 1 1 R N 397.235 -0.3781 -9.518295
+ 1 2022.03.31 MQC 2024.07.04D09:30:02.851000000 254.8359 258 SELL venue3 254.32 254.99 2 1 R V 254.655 0.1809 7.103729
+ 2 2022.03.31 BKR 2024.07.04D09:30:03.426000000 93.64192 154 BUY venue2 93.64 93.72 1 1 R N 93.68 0.03808 4.064902
+ 3 2022.03.31 GE.PRA 2024.07.04D09:30:04.489000000 397.5469 12 SELL venue2 397.21 398.8 1 2 R T 398.005 -0.4581 -11.50991
+ 4 2022.03.31 DXGE 2024.07.04D09:30:06.677000000 570.1041 60 SELL venue3 569.08 571.34 2 3 R P 570.21 -0.1059 -1.857211
+ ```
+
+## 2. Multi node workflow
+
+A multi node (distributed) compute consists of one driver node and one or more worker nodes.
+
+The driver node maintains state information of all notebooks attached to the compute, maintains the SparkContext, interprets all the commands you run from a notebook or a library on the compute, and runs the Apache Spark master that coordinates with the Spark executors.
+Worker nodes run the Spark executors and other services required for proper functioning compute.
+
+Use multi-node compute for larger jobs with distributed workloads.
+
+Before you start, make sure you meet the pre-requisites at the top of this page.
+
+### Import required libraries
+
+Import a PySpark and Python libraries by running:
+
+```python
+import pykx as kx
+from pyspark.sql import functions as F
+from pyspark.sql import SparkSession
+from pyspark.sql.functions import pandas_udf, PandasUDFType, lit
+from pyspark.sql.types import StructType, StructField, DoubleType, StringType, DateType, IntegerType
+spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", True)
+import copy as cp
+```
+
+### Prepare the data
+
+!!! note "The below examples will make use of the full dataset while using PySpark and a subset of the data while using the Python Driver."
+
+Create `trade_sdf_all_syms` and `quote_sdf_all_syms` Spark Dataframes containing trade and quote data respectively using Spark SQL:
+
+```python
+trade_sdf_all_syms = spark.sql('SELECT TTime as time, Symbol as sym, cast(Trade_Volume AS INT) as size, Trade_Price as price FROM default.nyse_trades')
+quote_sdf_all_syms = spark.sql('SELECT TTime as time, Symbol as sym, Bid_Size as bsize, Bid_Price as bid, Offer_Size as asize, Offer_Price as ask FROM default.nyse_quotes')
+```
+
+Filter the data to the a subset of the tickers:
+
+```python
+SYMBOLS = ['GOOG', 'AAPL', 'FB', 'AMZN', 'MSFT']
+trade_sdf_few_syms = trade_sdf_all_syms.where(F.col("Symbol").isin(SYMBOLS)).orderBy(["Symbol","TTime"])
+quote_sdf_few_syms = quote_sdf_all_syms.where(F.col("Symbol").isin(SYMBOLS)).orderBy(["Symbol","TTime"])
+```
+
+Convert the Spark Dataframes to Pandas Dataframes:
+
+```python
+trade_pdf = trade_sdf_few_syms.toPandas()
+quote_pdf = quote_sdf_few_syms.toPandas()
+```
+
+### Slippage markouts (Python Driver/Single Node)
+
+!!! note "Slippage - The difference between the expected price of a trade and the price at which the trade is executed."
+
+In the below example, we will use PyKX to calculate the execution slippage markouts for a subset of the data (5 tickers) using just the Python Driver (Single node).
+
+??? example "Define the required functions used to calculate the slippage markouts"
+
+ ```python
+ # adjust the time column in the past/future by offset e.g 10 20 30 seconds, 1 5 10 minutes
+ def time_offset(trade, offset):
+ offset_trade = cp.copy(trade)
+ offset_trade['time'] = (offset_trade['time'] + offset).sorted()
+ return offset_trade[['sym', 'time']]
+
+ # compute N markouts and calculate slippage using mid price
+ def trade_to_mid_markouts(trade, quote):
+ import os
+ os.environ['QLIC'] = '/dbfs/tmp/'
+ import pykx as kx
+
+ trade = kx.toq(trade)
+ quote = kx.toq(quote)
+
+ trade = trade.sort_values(['time']).grouped('sym')
+ quote = quote.sort_values(['time']).grouped('sym')
+ quote['mid'] = (quote['ask'] + quote['bid']) / 2
+
+ # markouts we are interested in
+ seconds = [1, 10, 30]
+ minutes = [1, 5, 10, 30]
+
+ qseconds = kx.toq(seconds).cast(kx.SecondVector)
+ qminutes = kx.toq(minutes).cast(kx.MinuteVector)
+
+ for s, q in zip(seconds, qseconds):
+ trade['tp'+str(s)+'s'] = trade['price'] - time_offset(trade, q).merge_asof(quote[['sym', 'time', 'mid']], on=['sym', 'time'])['mid']
+ trade['tm'+str(s)+'s'] = trade['price'] - time_offset(trade, -q).merge_asof(quote[['sym', 'time', 'mid']], on=['sym', 'time'])['mid']
+
+ for s, q in zip(minutes, qminutes):
+ trade['tp'+str(s)+'m'] = trade['price'] - time_offset(trade, q).merge_asof(quote[['sym', 'time', 'mid']], on=['sym', 'time'])['mid']
+ trade['tm'+str(s)+'m'] = trade['price'] - time_offset(trade, q).merge_asof(quote[['sym', 'time', 'mid']], on=['sym', 'time'])['mid']
+
+ return trade.pd()
+ ```
+
+Pass the `trade_pdf` and `quote_pdf` Pandas Dataframes to the `trade_to_mid_markouts` function in order to calculate the slippage:
+
+```python
+result = trade_to_mid_markouts(trade_pdf, quote_pdf)
+
+print('Executions input row count - ', f'{trade_pdf.shape[0]:,}')
+print('Market quotes input row count - ', f'{quote_pdf.shape[0]:,}')
+print('TCA report result row count - ', f'{kx.q.count(result).np():,}')
+
+result.tail()
+```
+
+!!! note "Since we have supplied a Pandas dataframe, the workload will be performed by only the Python driver."
+
+??? example "Output"
+
+ ```python
+ Executions input row count - 1,851,406
+ Market quotes input row count - 27,424,147
+ TCA report result row count - 1,851,406
+
+ time sym size price tp1s tm1s tp10s tm10s tp30s tm30s tp1m tm1m tp5m tm5m tp10m tm10m tp30m tm30m
+ 1851401 2022-10-03 19:59:55.559408 AAPL 1 142.935 0.000 0.000 -0.025 0.010 -0.025 0.045 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025
+ 1851402 2022-10-03 19:59:58.444119 MSFT 25 241.700 0.185 0.185 0.185 0.185 0.185 3.200 0.185 0.185 0.185 0.185 0.185 0.185 0.185 0.185
+ 1851403 2022-10-03 19:59:59.146947 AAPL 1 142.935 -0.025 0.000 -0.025 0.010 -0.025 0.055 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025 -0.025
+ 1851404 2022-10-03 19:59:59.551589 MSFT 1 241.700 0.185 0.185 0.185 0.185 0.185 3.200 0.185 0.185 0.185 0.185 0.185 0.185 0.185 0.185
+ 1851405 2022-10-03 19:59:59.668143 MSFT 150 241.700 0.185 0.185 0.185 0.185 0.185 3.200 0.185 0.185 0.185 0.185 0.185 0.185 0.185 0.1815
+ ```
+
+### Slippage markouts (PySpark/Multi node)
+
+In the below example, we will use PyKX to calculate the execution slippage markouts for all of the data (11686 tickers) using PySpark (Multi node).
+
+Prepare the schema for the results dataframe:
+
+```python
+full_schema = "time timestamp, sym string, size int, price double"
+
+seconds = [1, 10, 30]
+minutes = [1, 5, 10, 30]
+
+for s in seconds:
+ full_schema += ', tp'+str(s)+'s float'
+ full_schema += ', tm'+str(s)+'s float'
+for m in minutes:
+ full_schema += ', tp'+str(m)+'m float'
+ full_schema += ', tm'+str(m)+'m float'
+```
+
+Pass the `trade_sdf_all_syms` and `quote_sdf_all_syms` Spark Dataframes to the `trade_to_mid_markouts` function in order to calculate the slippage:
+
+```python
+result_pyspark = trade_sdf_all_syms.groupby('sym').cogroup(quote_sdf_all_syms.groupby('sym')).applyInPandas(trade_to_mid_markouts, schema=full_schema)
+
+print('Executions input row count - ', f'{trade_sdf_all_syms.count():,}')
+print('Market quotes input row count - ', f'{quote_sdf_all_syms.count():,}')
+print('TCA report result row count - ', f'{result_pyspark.count():,}')
+
+result_pyspark.show(5)
+```
+
+!!! note "Since we have supplied a Spark dataframe, the workload will be farmed out to the configured worker nodes for processing before returning the result to the Python driver."
+
+??? example "Output"
+
+ ```python
+ Executions input row count - 77,026,358
+ Market quotes input row count - 1,541,781,424
+ TCA report result row count - 77,026,358
+
+ +--------------------+------+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+-----+-----+
+ | time| sym|size|price| tp1s| tm1s|tp10s|tm10s|tp30s|tm30s| tp1m| tm1m| tp5m| tm5m| tp10m| tm10m|tp30m|tm30m|
+ +--------------------+------+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+-----+-----+
+ |2022-10-03 09:16:...|AKIC U| 85| 9.95| 0.21| 0.21| 0.21| 0.21| 0.21| 0.21| 0.21| 0.21| 0.17| 0.17|-0.265|-0.265|-0.03|-0.03|
+ |2022-10-03 11:20:...|AKIC U| 9| 9.96| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0|
+ |2022-10-03 11:56:...|AKIC U| 21| 9.96| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -1.0| -0.02| -0.02|-0.02|-0.02|
+ |2022-10-03 13:36:...|AKIC U| 21| 9.95|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01| -0.01| -0.01|-0.01|-0.01|
+ |2022-10-03 13:36:...|AKIC U| 21| 9.95|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01|-0.01| -0.01| -0.01|-0.01|-0.01|
+ +--------------------+------+----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+------+------+-----+-----+
+ ```
\ No newline at end of file
diff --git a/docs_archive/kx_for_databricks/getStarted.md b/docs_archive/kx_for_databricks/getStarted.md
new file mode 100644
index 0000000..4d0654c
--- /dev/null
+++ b/docs_archive/kx_for_databricks/getStarted.md
@@ -0,0 +1,228 @@
+---
+title: Get started with KX for Databricks
+description: How to install and use KX for Databricks
+date: September 2024
+author: KX Systems, Inc.,
+tags: PyKX, setup, install,
+---
+
+# Get started with KX for Databricks
+
+_This page explains how to deploy and get started with KX for Databricks._
+
+1. [Install PyKX](#1-install-pykx)
+2. [Licensing](#2-licensing)
+3. [Load data using Spark Dataframes](#3-load-data-using-spark-dataframes)
+4. [PyKX pythonic vs. q magic (%%)](#4-pykx-pythonic-vs-q-magic-)
+
+
+## 1. Install PyKX
+
+Install PyKX on the cluster using one of the following methods:
+
+!!! Note ""
+
+ === "Install PyKX from PyPI"
+
+ Create a new notebook and install using `pip`:
+
+ ```python
+ pip install pykx
+ ```
+
+ === "Install PyKX from Anaconda"
+
+ Create a new notebook and install using `conda`:
+
+ ```python
+ conda install -c kx pykx
+ ```
+
+ === "Install PyKX from the Python Wheel"
+
+ Download the Python Wheel file from the [Download Portal](https://portal.dl.kx.com/assets/pypi/pykx) and store it on the cluster:
+
+ ```shell
+ > ls /path/to/folder
+ pykx-2.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ ```
+
+ Create a new notebook and install from the wheel file:
+
+ ```python
+ pip install /path/to/folder/pykx-2.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ ```
+!!! tip "Tip: After installing the packages, you may need to restart the kernel by running the following command:"
+
+
+ ```python
+ dbutils.library.restartPython()
+ ```
+
+## 2. Licensing
+
+To use all PyKX functionalities, you need to download and install a KDB Insights license.
+
+1. Choose a license type:
+
+ !!! Note ""
+
+ === "Personal License"
+
+ For personal usage, navigate to [Personal License Download](https://kx.com/kdb-insights-personal-edition-license-download/) and complete the form.
+
+ === "Commercial License"
+
+ For commercial usage, contact your KX sales representative, email [sales@kx.com](mailto:sales@kx.com), or apply through [Book demo](https://kx.com/book-demo).
+
+
+ !!! info "More on the [license installation process](https://code.kx.com/pykx/2.5/getting-started/installing.html#2-install-a-kdb-insights-license)."
+
+2. On receipt of an email from KX, download and save the `kc.lic` license file to a secure location on your cluster:
+
+ ```shell
+ > ls /path/to/folder
+ kc.lic
+ ```
+
+3. Set an environment variable `QLIC` pointing to the folder containing the license file:
+
+ ```python
+ import os
+ os.environ['QLIC'] = '/path/to/folder'
+ ```
+ That's it! To verify if you successfully installed PyKX, run the following:
+
+ ```python
+ import pykx as kx
+ print('PyKX Version: ', kx.__version__)
+ ```
+ You should have the following output:
+
+ ```python
+ PyKX Version: 2.5.1
+ ```
+ !!! note "Note: The license requires internet connectivity."
+
+ Update the workspace security policies to whitelist the following:
+
+ - The remote servers should treat kdb+ on-demand as a CDN-fronted service and employ DNS-based filtering.
+ - The current list of hostnames required for kdb+ on-demand operation is {h,g}.kdbod.{com,net,org}.
+
+## 3. Load data using Spark Dataframes
+
+**a. From the Databricks Unity Catalogue:**
+
+```python
+dfTrades = spark.table('..')
+dfQuotes = spark.table('..')
+```
+
+**b. Directly from a CSV file:**
+
+```python
+dfTrades = spark.read.format("csv").option("header", "true").option("delimiter", ",").load("/path/to/file")
+dfQuotes = spark.read.format("csv").option("header", "true").option("delimiter", ",").load("/path/to/file")
+```
+
+To convert the Spark Dataframe to a Pandas Dataframe, use the `toPandas()` function:
+
+```python
+pandas_trade = dfTrades.toPandas()
+pandas_quote = dfQuotes.toPandas()
+```
+
+Convert the Pandas Dataframe to a q table with the `toq()` function:
+
+```python
+pykx_trades = kx.toq(pandas_trade)
+pykx_quotes = kx.toq(pandas_quote)
+```
+
+## 4. PyKX pythonic vs. q magic (%%)
+
+Reference q functions in PyKX with the standard python syntax:
+
+```python
+kx.q('til 10')
+```
+
+This produces a sequence of numbers from 0 to 9:
+
+```python
+pykx.LongVector(pykx.q('0 1 2 3 4 5 6 7 8 9'))
+```
+
+Another way to achieve this is by using native q by leveraging q magic (`%%q`) :
+
+```python
+%%q
+til 10
+```
+
+You will get the same output:
+
+```python
+0 1 2 3 4 5 6 7 8 9
+```
+
+??? example "Simple load and query example using PyKX"
+
+ Consider a `samples` catalogue with a `finance` schema that has a `trades` table. Load data from the table into a Spark Dataframe `dfTrade`:
+
+ ```python
+ dfTrade = spark.table('samples.finance.trades')
+ ```
+
+ Next, convert the Spark Dataframe to a Pandas Dataframe `panda_trade` using the `toPandas()` function:
+
+ ```python
+ panda_trade = dfTrade.toPandas()
+ ```
+
+ Using the `toq()` function to convert the Pandas Dataframe to a q table `pykx_trade`:
+
+ ```python
+ pykx_trade = kx.toq(panda_trade)
+ ```
+
+ To perform a simple OHLC query on the data, use regular python syntax:
+
+ ```python
+ kx.q("{select Open:first price,High:max price,Low:min price,Close:last price by bucket:0D01 xbar time,sym from x}", pykx_trade).head()
+ ```
+
+ This should produce the following output:
+
+ ```python
+ bucket sym Open High Low Close
+ 2017.08.31D00:00:00.000000000 A 347.8271 349.256 346.0935 348.2853
+ AAN 347.4742 349.1463 346.2195 348.3558
+ ACA 347.4005 348.4313 346.5526 347.2568
+ ACP 346.8897 350.9313 345.8342 347.8243
+ ```
+
+ You can achieve the same result with q magic by storing the table:
+
+ ```python
+ kx.q['pykx_trade'] = pykx_trade
+ ```
+
+ and performing the same query:
+
+ ```python
+ %%q
+ select Open:first price,High:max price,Low:min price,Close:last price by bucket:0D01 xbar time,sym from pykx_trade
+ ```
+
+ In the end, you get the same output:
+
+ ```python
+ bucket sym | Open High Low Close
+ ----------------------------------| -----------------------------------
+ 2017.08.31D00:00:00.000000000 A | 347.8271 349.256 346.0935 348.2853
+ 2017.08.31D00:00:00.000000000 AAN | 347.4742 349.1463 346.2195 348.3558
+ 2017.08.31D00:00:00.000000000 ACA | 347.4005 348.4313 346.5526 347.2568
+ 2017.08.31D00:00:00.000000000 ACP | 346.8897 350.9313 345.8342 347.8243
+ 2017.08.31D00:00:00.000000000 ACRE| 349.8609 351.621 345.0562 345.9471
+ ```
\ No newline at end of file
diff --git a/docs_archive/kx_for_databricks/introduction.md b/docs_archive/kx_for_databricks/introduction.md
new file mode 100644
index 0000000..f9f1149
--- /dev/null
+++ b/docs_archive/kx_for_databricks/introduction.md
@@ -0,0 +1,35 @@
+---
+title: KX for Databricks
+description: Kx, PyKX, and Databricks
+date: September 2024
+author: KX Systems, Inc.,
+tags: kx, PyKX, Databricks
+---
+
+# KX for Databricks
+_This page briefly describes PyKX on Databricks use cases and capabilities._
+
+KX for Databricks brings the full capability of KX's time series engine to Databricks. KX's PyKX, the Python-first interface already familiar to data scientists and quants, turbocharges workloads and research on use cases such as:
+
+- Algorithmic trading strategy development and backtesting
+- Large-scale pre- and post-trade analytics
+- Macro-economic research and analysis using large datasets
+- Cross-asset analytics with deep historical data
+
+KX for Databricks leverages PyKX directly inside Databricks, for Python and distributed Spark workloads. Use it on existing Delta Lake datasets without external KX dependencies, or choose to flexibly interop with other KX products to leverage the power of your broader KX systems estate.
+
+
+
+## Get started and workflows
+
+Refer to the [Get started](getStarted.md) page if you wish to complete the following actions:
+
+- [Install](getStarted.md#1-install-pykx) PyKX on Databricks.
+- Check what type of [license](getStarted.md#2-licensing) you need.
+- [Load data](getStarted.md#3-load-data-using-spark-dataframes) using Spark Dataframes.
+- [Use PyKX pythonic vs. q magic (%%)](getStarted.md#4-pykx-pythonic-vs-q-magic-).
+
+Next, learn how to [set up a cluster](clusterSetups.md), depending on the type of workflow you need:
+
+- Work with a [single node/driver](clusterSetups.md#1-single-node-workflow) on analytic calculation of OHLC, VWAP pricing, volatility, spread, and trade execution analysis (slippage).
+- Work with a [multi/distributed nodes](clusterSetups.md#2-multi-node-workflow) cluster on slippage calculation with PySpark.
diff --git a/docs_archive/kx_for_databricks/kx-databricks-refarch.png b/docs_archive/kx_for_databricks/kx-databricks-refarch.png
new file mode 100644
index 0000000..f24db3b
Binary files /dev/null and b/docs_archive/kx_for_databricks/kx-databricks-refarch.png differ
diff --git a/mkdocs.yml b/mkdocs.yml
index ab75569..6e2f759 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -28,7 +28,7 @@ extra_css:
- https://code.kx.com/dashboards/stylesheets/dashboards.css
- stylesheets/swagger.css
# More accessible colors in the edited versions
- - stylesheets/2021-edited.css
+ - stylesheets/2026-edited.css
- stylesheets/prism-edited.css
extra_javascript:
@@ -40,6 +40,7 @@ extra_javascript:
- 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML'
- 'https://code.jquery.com/jquery-3.6.0.min.js'
- 'https://use.fontawesome.com/3dbf29ed12.js'
+ - scripts/strip-q-prompt-copy.js
# this relative path is needed to source from code.kx.com!!!
- "../../pykx/scripts/versions.js"
# Uncomment only for local test
@@ -181,125 +182,118 @@ theme:
docs_dir: docs
nav:
- - Home: 'https://code.kx.com/insights/'
- - kdb+ and q: 'https://code.kx.com/q'
- - kdb Insights SDK: "https://code.kx.com/insights/core"
- - kdb Insights Enterprise: "https://docs.kx.com/insights-enterprise-latest/insights/enterprise/home.htm"
- - KDB.AI: "https://code.kx.com/kdbai/latest/"
- - PyKX:
- - index.md
+ - Home:
+ - Overview: index.md
- Get Started:
- Install: getting-started/installing.md
- Quickstart guide: getting-started/quickstart.md
- - Learn:
- - What is PyKX?: getting-started/what_is_pykx.md
- - Objects and attributes: learn/objects.md
- - Databases: user-guide/advanced/database/index.md
- - Modes of operation: user-guide/advanced/modes.md
- - Performance tips: user-guide/advanced/performance.md
- - Why upgrade from embedPy: pykx-under-q/upgrade.md
- - Glossary: extras/glossary.md
- - Blogs, articles, videos: blogs.md
- - How to:
- - Configure PyKX: user-guide/configuration.md
- - Interact with data:
- - Create objects: user-guide/fundamentals/creating.md
- - Use objects: user-guide/fundamentals/evaluating.md
- - Index objects: user-guide/fundamentals/indexing.md
- - Convert data: user-guide/fundamentals/conversion_considerations.md
- - Convert text: user-guide/fundamentals/text.md
- - Convert nulls and infinities: user-guide/fundamentals/nulls_and_infinities.md
- - Convert temporal data: user-guide/fundamentals/temporal.md
- - Interact with databases:
- - Generate databases: user-guide/advanced/database/db_gen.md
- - Load databases: user-guide/advanced/database/db_loading.md
- - Manage databases: user-guide/advanced/database/db_mgmt.md
- - Query data:
- - Introduction: user-guide/fundamentals/query/index.md
- - Query with Python: user-guide/fundamentals/query/pyquery.md
- - Query with SQL: user-guide/fundamentals/query/sql.md
- - Query with q: user-guide/fundamentals/query/qquery.md
- - Performance considerations: user-guide/fundamentals/query/perf.md
- - Communicate via IPC: user-guide/advanced/ipc.md
- - Capture real-time data:
- - Introduction: user-guide/advanced/streaming/index.md
- - Capture and Store: user-guide/advanced/streaming/basic.md
- - Publish data: user-guide/advanced/streaming/publish.md
- - Subscribe to data: user-guide/advanced/streaming/subscribe.md
- - Analyze streaming data: user-guide/advanced/streaming/rta.md
- - Build custom APIs: user-guide/advanced/streaming/custom_apis.md
- - Manage query routing: user-guide/advanced/streaming/gateways.md
- - Complex infrastructure: user-guide/advanced/streaming/complex.md
- - Apply attributes: user-guide/advanced/attributes.md
- - Compress and encrypt data: user-guide/advanced/compress-encrypt.md
- - Import existing q functions: user-guide/advanced/context_interface.md
- - Run remote functions: user-guide/advanced/remote-functions.md
- - Serialize data: user-guide/advanced/serialization.md
- - Use PyKX in subprocesses: user-guide/advanced/subprocess.md
- - Run q code multithreaded: user-guide/advanced/threading.md
- - Manage your license: user-guide/advanced/license.md
- - Use Python in a q process: pykx-under-q/intro.md
- - Reference:
- - API references:
- - q functions and operators: api/pykx-execution/q.md
- - Execution classes: api/pykx-execution/embedded_q.md
- - q contexts: api/pykx-execution/ctx.md
- - Emulated q console: api/pykx-execution/console.md
- - Data generation: api/random.md
- - Schema generation: api/schema.md
- - Query data: api/query.md
- - Query classes: api/columns.md
- - Registering custom operations: api/pykx-q-data/register.md
- - Compression and encryption: api/compress.md
- - Database management: api/db.md
- - File I/O: api/pykx-save-load/fileio.md
- - q IPC interface: api/ipc.md
- - Streamlit integration: api/streamlit.md
- - Serialize/deserialize: api/serialize.md
- - Remote Python execution: api/remote.md
- - Real-time capture: api/tick.md
- - System command wrappers: api/system.md
- - Utility functions: api/util.md
- - Reimporter module: api/reimporting.md
- - License management: api/license.md
- - PyKX under q: pykx-under-q/api.md
- - Data types and conversions:
- - Python to PyKX: api/pykx-q-data/toq.md
- - Wrappers: api/pykx-q-data/wrappers.md
- - PyKX to Python: api/pykx-q-data/type_conversions.md
- - Pandas API: user-guide/advanced/Pandas_API.ipynb
- - Exceptions: api/exceptions.md
- - Integrations:
- - NumPy: user-guide/advanced/numpy.md
- - Streamlit: user-guide/advanced/streamlit.md
- - Python charting libraries: examples/charting.ipynb
- - Jupyter Integration: examples/jupyter-integration.ipynb
- - Examples:
- - PyKX Introduction Notebook: examples/interface-overview.ipynb
- - Subscriber: examples/subscriber/readme.md
- - Compress and encrypt: examples/compress_and_encrypt/readme.md
- - Database creation and management: examples/db-management.ipynb
- - IPC: examples/ipc/README.md
- - PyKX as a server: examples/server/server.md
- - Real-time streaming: examples/streaming/index.md
- - Multithreaded execution: examples/threaded_execution/threading.md
- - Asynchronous Queries: examples/AsynchronousQueries/async_querying.md
- - Releases:
- - Release notes:
- - PyKX: release-notes/changelog.md
- - PyKX under q: release-notes/underq-changelog.md
- - Deprecations: release-notes/deprecations.md
- - 2.x -> 3.x Upgrade : upgrades/2030.md
- - Beta features:
- - Introduction: beta-features/index.md
- - PyTorch Conversions: beta-features/torch.md
- - Help and Support:
- - Troubleshooting: help/troubleshooting.md
- - FAQ: help/faq.md
- - PyKX support: help/support.md
- - Issues and limitations: help/issues.md
- - PyKX license: license.md
- - Contributors: contributors.md
- - APIs: 'https://code.kx.com/insights/api/index.html'
- - Licensing: 'https://code.kx.com/insights/licensing/index.html'
- - Help: 'https://code.kx.com/home/support.html'
+ - Learn:
+ - What is PyKX?: getting-started/what_is_pykx.md
+ - Objects and attributes: learn/objects.md
+ - Databases: user-guide/advanced/database/index.md
+ - Modes of operation: user-guide/advanced/modes.md
+ - Performance tips: user-guide/advanced/performance.md
+ - Why upgrade from embedPy: pykx-under-q/upgrade.md
+ - Glossary: extras/glossary.md
+ - Blogs, articles, videos: blogs.md
+ - How to:
+ - Configure PyKX: user-guide/configuration.md
+ - Interact with data:
+ - Create objects: user-guide/fundamentals/creating.md
+ - Use objects: user-guide/fundamentals/evaluating.md
+ - Index objects: user-guide/fundamentals/indexing.md
+ - Convert data: user-guide/fundamentals/conversion_considerations.md
+ - Convert text: user-guide/fundamentals/text.md
+ - Convert nulls and infinities: user-guide/fundamentals/nulls_and_infinities.md
+ - Convert temporal data: user-guide/fundamentals/temporal.md
+ - Interact with databases:
+ - Generate databases: user-guide/advanced/database/db_gen.md
+ - Load databases: user-guide/advanced/database/db_loading.md
+ - Manage databases: user-guide/advanced/database/db_mgmt.md
+ - Query data:
+ - Introduction: user-guide/fundamentals/query/index.md
+ - Query with Python: user-guide/fundamentals/query/pyquery.md
+ - Query with SQL: user-guide/fundamentals/query/sql.md
+ - Query with q: user-guide/fundamentals/query/qquery.md
+ - Performance considerations: user-guide/fundamentals/query/perf.md
+ - Communicate via IPC: user-guide/advanced/ipc.md
+ - Capture real-time data:
+ - Introduction: user-guide/advanced/streaming/index.md
+ - Capture and Store: user-guide/advanced/streaming/basic.md
+ - Publish data: user-guide/advanced/streaming/publish.md
+ - Subscribe to data: user-guide/advanced/streaming/subscribe.md
+ - Analyze streaming data: user-guide/advanced/streaming/rta.md
+ - Build custom APIs: user-guide/advanced/streaming/custom_apis.md
+ - Manage query routing: user-guide/advanced/streaming/gateways.md
+ - Complex infrastructure: user-guide/advanced/streaming/complex.md
+ - Apply attributes: user-guide/advanced/attributes.md
+ - Compress and encrypt data: user-guide/advanced/compress-encrypt.md
+ - Import existing q functions: user-guide/advanced/context_interface.md
+ - Run remote functions: user-guide/advanced/remote-functions.md
+ - Serialize data: user-guide/advanced/serialization.md
+ - Use PyKX in subprocesses: user-guide/advanced/subprocess.md
+ - Run q code multithreaded: user-guide/advanced/threading.md
+ - Manage your license: user-guide/advanced/license.md
+ - Use Python in a q process: pykx-under-q/intro.md
+ - Reference:
+ - API references:
+ - q functions and operators: api/pykx-execution/q.md
+ - Execution classes: api/pykx-execution/embedded_q.md
+ - q contexts: api/pykx-execution/ctx.md
+ - Emulated q console: api/pykx-execution/console.md
+ - Data generation: api/random.md
+ - Schema generation: api/schema.md
+ - Query data: api/query.md
+ - Query classes: api/columns.md
+ - Registering custom operations: api/pykx-q-data/register.md
+ - Compression and encryption: api/compress.md
+ - Database management: api/db.md
+ - File I/O: api/pykx-save-load/fileio.md
+ - q IPC interface: api/ipc.md
+ - Streamlit integration: api/streamlit.md
+ - Serialize/deserialize: api/serialize.md
+ - Remote Python execution: api/remote.md
+ - Real-time capture: api/tick.md
+ - System command wrappers: api/system.md
+ - Utility functions: api/util.md
+ - Reimporter module: api/reimporting.md
+ - License management: api/license.md
+ - PyKX under q: pykx-under-q/api.md
+ - Data types and conversions:
+ - Python to PyKX: api/pykx-q-data/toq.md
+ - Wrappers: api/pykx-q-data/wrappers.md
+ - PyKX to Python: api/pykx-q-data/type_conversions.md
+ - Pandas API: user-guide/advanced/Pandas_API.ipynb
+ - Exceptions: api/exceptions.md
+ - Integrations:
+ - NumPy: user-guide/advanced/numpy.md
+ - Streamlit: user-guide/advanced/streamlit.md
+ - Python charting libraries: examples/charting.ipynb
+ - Jupyter Integration: examples/jupyter-integration.ipynb
+ - Examples:
+ - PyKX Introduction Notebook: examples/interface-overview.ipynb
+ - Subscriber: examples/subscriber/readme.md
+ - Compress and encrypt: examples/compress_and_encrypt/readme.md
+ - Database creation and management: examples/db-management.ipynb
+ - IPC: examples/ipc/README.md
+ - PyKX as a server: examples/server/server.md
+ - Real-time streaming: examples/streaming/index.md
+ - Multithreaded execution: examples/threaded_execution/threading.md
+ - Asynchronous Queries: examples/AsynchronousQueries/async_querying.md
+ - Releases:
+ - Release notes:
+ - PyKX: release-notes/changelog.md
+ - PyKX under q: release-notes/underq-changelog.md
+ - Deprecations: release-notes/deprecations.md
+ - 2.x -> 3.x Upgrade : upgrades/2030.md
+ - Beta features:
+ - Introduction: beta-features/index.md
+ - PyTorch Conversions: beta-features/torch.md
+ - Help and Support:
+ - Troubleshooting: help/troubleshooting.md
+ - FAQ: help/faq.md
+ - PyKX support: help/support.md
+ - Issues and limitations: help/issues.md
+ - PyKX license: license.md
+ - Contributors: contributors.md
+
diff --git a/src/pykx/_wrappers.pyx b/src/pykx/_wrappers.pyx
index a0913ce..8510220 100644
--- a/src/pykx/_wrappers.pyx
+++ b/src/pykx/_wrappers.pyx
@@ -60,6 +60,14 @@ NPY_TYPE_ITEMSIZE = {
}
+def memview_to_array(view) -> np.ndarray:
+ cdef np.npy_intp n = (view._ptr).n
+ cdef np.ndarray arr = np.PyArray_SimpleNewFromData(1, &n, 2, (view._ptr).G0)
+ Py_INCREF(view)
+ PyArray_SetBaseObject(arr, view)
+ return arr
+
+
def k_vec_to_array(k_vec, np_type: int) -> np.ndarray:
"""Creates a Numpy array that references the K object until the array's data is deallocated."""
cdef np.npy_intp n = (k_vec._addr).n
@@ -118,7 +126,11 @@ def k_str(self):
cpdef inline deserialize(x):
- cdef core.K buff = core.kpn(x, len(x))
+ l = len(x)
+ if isinstance(x, np.ndarray):
+ x = x.data.tobytes()
+ cdef core.K buff = core.kpn(x, l)
+
if 0 == core.okx(buff):
core.r0(buff)
raise QError('Failed to deserialize supplied non PyKX IPC serialized format object')
diff --git a/src/pykx/lib/4-1-libs/q.k b/src/pykx/lib/4-1-libs/q.k
index 74d1eb4..b48fde4 100644
--- a/src/pykx/lib/4-1-libs/q.k
+++ b/src/pykx/lib/4-1-libs/q.k
@@ -117,11 +117,11 @@ IN:{$[99h<@x;x in y;0]};qa:{$[qb x;0;IN[*x;a0];1;|/qa'1_x]};qb:{(2>#x)|(@x)&~11=
/ CAN EXIT HERE FOR SMALL Q
/ pt(tables) pf(date/month/year/int) pd(dirs) pv(values) pn(count) pt::0#pf::`
vt:(,`)!,()!();
-bvfp:{g:$[(::)~x;max;min];x:.Q.d;d:{$[y~pv;`/:'x,'d@&(d:!x)in`$$y;u@&{11h=@!x}'u:`/:'x,'`$$y]}[;y@:&y in pv]'P:$[`par.txt in!x;jp[x]'`$0:`/:x,`par.txt;,x];if[.Q.vt~(,`)!,()!();.Q.vt:P!(#P)#.:.Q.vt];
+bvfp:{g:$[(::)~x;max;min];x:.Q.d;d:$[y~z;{`/:'x,'d@&(d:!x)in`$$y}[;z];{u@&{11h=@!x}'u:`/:'x,'`$$y}[;y@:&y in pv]]'P:$[`par.txt in!x;jp[x]'`$0:`/:x,`par.txt;,x];if[.Q.vt~(,`)!,()!();.Q.vt:P!(#P)#.:.Q.vt];
t:?,/!:'vt:{(&#:'x)(=,/. x)}'{({("DMJJ"`date`month`year`int?pf)$$last@`\:x}'x)!!:':x}'d;
- if[#nt:(exc:.q.except)[t;!vp];d:{`/:'x[(. y)[;0]],'(`$$(. y)[;1]),'!y}[P]@{i:y@&:x=y x:@[x;&x~\:();:;*0#pv];(i;x i)}[;g]'+:nt#/:g''nt#/:vt;.Q.vp,:nt!{(+(,pf)!,0#pv),'+(-2!'.+x)#'+|0#x:?[x;();0b;()]}':d;.Q.vt:.Q.vt,\:nt!(#nt)#,exc[?pv;y]];.Q.vt:(.Q.vt,''P!exc[y]''vt)exc''vt:(nt,!vp)#/:vt;{.[x;();:;+.q.except[!+vp x;pf]!x]}'exc[!vp;pt];pt::.q.asc@!vp;vpv::?pv;}
-bvi:{$[(`vpv in !.Q);bvfp[x;?.q.except[pv;vpv]];bv x]}
-bv:{vt::(,`)!,()!();vp::(0#`)!();vpv::0#pv;bvfp[x;?pv]}
+ if[#nt:(exc:.q.except)[t;!vp];d:{`/:'x[(. y)[;0]],'(`$$(. y)[;1]),'!y}[P]@{i:y@&:x=y x:@[x;&x~\:();:;*0#pv];(i;x i)}[;g]'+:nt#/:g''nt#/:vt;.Q.vp,:nt!{(+(,pf)!,0#pv),'+(-2!'.+x)#'+|0#x:?[x;();0b;()]}':d;.Q.vt:.Q.vt,\:nt!(#nt)#,exc[z;y]];.Q.vt:(.Q.vt,''P!exc[y]''vt)exc''vt:(nt,!vp)#/:vt;{.[x;();:;+.q.except[!+vp x;pf]!x]}'exc[!vp;pt];pt::.q.asc@!vp;vpv::z;}
+bvi:{$[(`vpv in !.Q);bvfp[x;?.q.except[pv;vpv];?pv];bv x]}
+bv:{vt::(,`)!,()!();vp::(0#`)!();vpv::0#pv;dpv:?pv;bvfp[x;dpv;dpv]}
sp:{$[0>."\\s";x'y;x':y]}
pt:pm:();MAP:{{$[0>@a:.+0!. x;.q.set[x]@.`$-1_$a;]}'a@&~(a:."\\a")in pt;pm::();if[#pt;pm::pt!sp[{(`u#pd,'pv)!sp[p2[(x;();0b;())]/;+(pd;pv)]}]pt]}
diff --git a/src/pykx/q.so/libs/4-1/l64/libq.so b/src/pykx/q.so/libs/4-1/l64/libq.so
index 48ff188..56684db 100755
Binary files a/src/pykx/q.so/libs/4-1/l64/libq.so and b/src/pykx/q.so/libs/4-1/l64/libq.so differ
diff --git a/src/pykx/q.so/libs/4-1/l64arm/libq.so b/src/pykx/q.so/libs/4-1/l64arm/libq.so
index 3ea98ee..3e2e4f9 100755
Binary files a/src/pykx/q.so/libs/4-1/l64arm/libq.so and b/src/pykx/q.so/libs/4-1/l64arm/libq.so differ
diff --git a/src/pykx/q.so/libs/4-1/m64/libq.dylib b/src/pykx/q.so/libs/4-1/m64/libq.dylib
index 5d1f4c9..5358dba 100755
Binary files a/src/pykx/q.so/libs/4-1/m64/libq.dylib and b/src/pykx/q.so/libs/4-1/m64/libq.dylib differ
diff --git a/src/pykx/q.so/libs/4-1/m64arm/libq.dylib b/src/pykx/q.so/libs/4-1/m64arm/libq.dylib
index 86b67a2..21660d4 100755
Binary files a/src/pykx/q.so/libs/4-1/m64arm/libq.dylib and b/src/pykx/q.so/libs/4-1/m64arm/libq.dylib differ
diff --git a/src/pykx/q.so/libs/4-1/w64/q.dll b/src/pykx/q.so/libs/4-1/w64/q.dll
index eeee7f1..35c9677 100644
Binary files a/src/pykx/q.so/libs/4-1/w64/q.dll and b/src/pykx/q.so/libs/4-1/w64/q.dll differ
diff --git a/src/pykx/serialize.py b/src/pykx/serialize.py
index 183f1c5..45e3c58 100644
--- a/src/pykx/serialize.py
+++ b/src/pykx/serialize.py
@@ -3,7 +3,9 @@
"""
from typing import Any, Union
-from ._wrappers import _to_bytes, decref, deserialize as _deserialize
+import numpy as np
+
+from ._wrappers import _to_bytes, decref, deserialize as _deserialize, memview_to_array
from .wrappers import K
@@ -70,6 +72,28 @@ def __getitem__(self, key):
def __del__(self):
decref(self._ptr)
+ def np(self):
+ """Returns a copy of the bytes making up the serialized object as a NumPy array.
+
+ Examples:
+
+ Serializing a `K` object and then copying the serialized data to a new variable.
+
+ ```python
+ >>> k_obj = kx.q('til 10')
+ >>> ser = kx.serialize(k_obj)
+ >>> k_obj_array = ser.np()
+ >>> k_obj_array
+ array([ 1, 0, 0, 0, 94, 0, 0, 0, 7, 0, 10, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0,
+ 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0,
+ 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0,
+ 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0,
+ 0, 9, 0, 0, 0, 0, 0, 0, 0], dtype=uint8)
+ ```
+ """
+ return memview_to_array(self)
+
def copy(self):
"""Returns a copy of the bytes making up the serialized object.
@@ -120,4 +144,6 @@ def deserialize(data: Union[bytes, serialize, memoryview]):
return _deserialize(data.data.tobytes())
elif isinstance(data, memoryview):
return _deserialize(data.tobytes())
+ elif isinstance(data, np.ndarray):
+ return _deserialize(data)
return _deserialize(data)
diff --git a/src/pykx/wrappers.py b/src/pykx/wrappers.py
index 0459f53..bd1020d 100644
--- a/src/pykx/wrappers.py
+++ b/src/pykx/wrappers.py
@@ -11047,6 +11047,35 @@ def variance(self, sample=False, iterator=None):
fn = 'svar' if sample else 'var'
return self.call(fn, iterator=iterator)
+ def bin(self, other, iterator=None, col_arg_ind=1, project_args=None):
+ """
+ Perform a binary search on the Column.
+
+ Parameters:
+ other: A sorted list of same type as column.
+ iterator: What iterator to use when operating on the column
+ for example, to execute per row, use `each`.
+ col_arg_ind: Determines the index within the multivariate function
+ where the column parameter will be used. Default 1.
+ project_args: The argument indices of a multivariate function which will be
+ projected on the function before evocation with use of an iterator.
+
+ Example:
+
+ ```python
+ >>> import pykx as kx
+ >>> t = kx.q('([] n: 2 4)')
+ >>> t.select(kx.Column('n').bin([1,3,5]))
+ pykx.Table(pykx.q('
+ n
+ -
+ 0
+ 1
+ '))
+ """
+ return self.call('bin', other, iterator=iterator, col_arg_ind=col_arg_ind,
+ project_args=project_args)
+
class QueryPhrase:
"""Special wrapper for a list which will be treated as a QueryPhrase.
diff --git a/tests/test_pykx.py b/tests/test_pykx.py
index 5eacebb..22b744d 100644
--- a/tests/test_pykx.py
+++ b/tests/test_pykx.py
@@ -448,31 +448,31 @@ def test_kx_versions(kx):
if test_vars == ('Linux', 'x86_64', '4.0'):
assert kx.q.z.k == kx.q('2025.02.18')
elif test_vars == ('Linux', 'x86_64', '4.1'):
- assert kx.q.z.k == kx.q('2026.01.23')
+ assert kx.q.z.k == kx.q('2026.04.01')
# elif test_vars == ('Linux', 'x86_64', '4.2'):
# assert kx.q.z.k == kx.q('?')
elif test_vars == ('Linux', 'aarch64', '4.0'):
assert kx.q.z.k == kx.q('2025.02.18')
elif test_vars == ('Linux', 'aarch64', '4.1'):
- assert kx.q.z.k == kx.q('2026.01.23')
+ assert kx.q.z.k == kx.q('2026.04.01')
# elif test_vars == ('Linux', 'aarch64', '4.2'):
# assert kx.q.z.k == kx.q('?')
elif test_vars == ('Darwin', 'x86_64', '4.0'):
assert kx.q.z.k == kx.q('2025.02.18')
elif test_vars == ('Darwin', 'x86_64', '4.1'):
- assert kx.q.z.k == kx.q('2026.01.23')
+ assert kx.q.z.k == kx.q('2026.04.01')
# elif test_vars == 'Darwin', 'x86_64', '4.2'):
# assert kx.q.z.k == kx.q('?')
elif test_vars == ('Darwin', 'arm64', '4.0'):
assert kx.q.z.k == kx.q('2025.02.18')
elif test_vars == ('Darwin', 'arm64', '4.1'):
- assert kx.q.z.k == kx.q('2026.01.23')
+ assert kx.q.z.k == kx.q('2026.04.01')
# elif test_vars == ('Darwin', 'arm', '4.2'):
# assert kx.q.z.k == kx.q('?')
elif test_vars == ('Windows', 'AMD64', '4.0'):
assert kx.q.z.k == kx.q('2025.02.18')
elif test_vars == ('Windows', 'AMD64', '4.1'):
- assert kx.q.z.k == kx.q('2026.01.23')
+ assert kx.q.z.k == kx.q('2026.04.01')
# elif test_vars == ('Windows', 'AMD64', '4.2'):
# assert kx.q.z.k == kx.q('?')
else:
diff --git a/tests/test_util.py b/tests/test_util.py
index 16d3449..1d74be0 100644
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -5,6 +5,7 @@
from time import sleep
from uuid import uuid4
+import numpy as np
import pytest
import toml
@@ -241,3 +242,17 @@ def test_config_add_type(kx):
assert data['default']['PYKX_MAX_ERROR_LENGTH'] == 1
assert data['default']['PYKX_BETA_FEATURES']
os.remove(fpath)
+
+
+def test_serialize_deserialize(kx, q):
+ vec = q('til 10')
+ ser = kx.serialize(vec)
+ assert isinstance(ser, kx.serialize)
+
+ byte = ser.copy()
+ arr = ser.np()
+ assert isinstance(byte, bytes)
+ assert isinstance(arr, np.ndarray)
+
+ assert q('{x ~ til 10}', kx.deserialize(byte)).py()
+ assert q('{x ~ til 10}', kx.deserialize(arr)).py()
diff --git a/tests/test_wrappers.py b/tests/test_wrappers.py
index 3a4c2e4..4824067 100644
--- a/tests/test_wrappers.py
+++ b/tests/test_wrappers.py
@@ -5772,3 +5772,11 @@ def test_column_types(kx):
with pytest.raises(TypeError) as err:
kx.Column(age=5, data=(1, 2, 3))
assert "name can only be of type Str and pykx.SymbolAtom" in str(err)
+
+
+def test_column_bin(kx):
+ t = kx.q('([] n: 2 4)')
+ assert all(t.select(kx.Column('n').bin([1, 3, 5])) == kx.q('select 1 3 5 bin n from ([] n: 2 4)')) # noqa E:501
+
+ with pytest.raises(kx.QError):
+ t.select(kx.Column('n').bin(['a', 'b', 'c']))