当前是英文 | chinese version
"The fastest file I/O in Gensokyo, swift as the Wind God Maiden."
— Aya Shameimaru, always flying at full speed
Cross-platform asynchronous file I/O library using native async I/O where available.
Windows leverages IOCP (I/O Completion Ports), Linux uses io_uring (kernel 5.1+), and macOS uses Dispatch I/O (GCD) for truly non-blocking file operations.
see -> CHANGES
| Platform | Backend | True Async | Description |
|---|---|---|---|
| Windows | IOCP | ✅ | NT kernel native I/O Completion Ports |
| Linux | io_uring | ✅ | Next-gen async I/O (kernel 5.1+) |
| macOS | Dispatch I/O | ✅ | GCD kernel-level async I/O |
ayafileio aims to be a single, unified async file API that delivers true kernel-level async I/O across Windows, Linux, and macOS.
| Feature | Description |
|---|---|
| 🍃 Zero thread overhead | No background threads on true async platforms |
| 📰 Kernel-level completion | IOCP / io_uring / Dispatch I/O direct to kernel |
| ⚡ High concurrency | Handles thousands of concurrent file operations |
| 🎴 Familiar API | aiofiles-compatible, supports async/await |
| 📖 Text & binary support | Automatic encoding/decoding in text modes |
| 🔧 Unified configuration | Runtime tunable parameters for all backends |
| 🌍 Cross-platform | Windows, Linux, and macOS |
| 🐍 Latest Python | Supports 3.10–3.14, including 3.14t free-threading |
pip install ayafileioSystem requirements:
- Python 3.10+
- Windows 7+ / Linux (kernel 5.1+ for io_uring) / macOS 10.10+
- No external dependencies, precompiled wheels available
import asyncio
import ayafileio
async def main():
# Write to a file — fast as the wind
async with ayafileio.open("example.txt", "w") as f:
await f.write("Hello, async world!\n")
# Read with automatic decoding
async with ayafileio.open("example.txt", "r", encoding="utf-8") as f:
content = await f.read()
print(content)
# Binary operations
async with ayafileio.open("data.bin", "rb") as f:
data = await f.read(1024)
await f.seek(0, 0)
asyncio.run(main())ayafileio's file open/close overhead is already in the microsecond range, but for maximum performance, avoid reopening the same file in a loop.
# ❌ DO NOT DO THIS: repeated open/close in a loop
for i in range(10000):
async with ayafileio.open("data.bin", "rb") as f:
data = await f.read()
# ✅ DO THIS: open once, operate many times
async with ayafileio.open("data.bin", "rb") as f:
for i in range(10000):
await f.seek(0)
data = await f.read()The latter is ~6x faster — it eliminates 9999 unnecessary coroutine scheduling round-trips.
Check which backend is currently in use:
import ayafileio
info = ayafileio.get_backend_info()
print(info)
# Windows: {'platform': 'windows', 'backend': 'iocp', 'is_truly_async': True}
# Linux: {'platform': 'linux', 'backend': 'io_uring', 'is_truly_async': True}
# macOS: {'platform': 'macos', 'backend': 'dispatch_io', 'is_truly_async': True}ayafileio provides a unified configuration system that allows runtime tuning:
import ayafileio
# View current configuration
config = ayafileio.get_config()
print(config)
# Update configuration
ayafileio.configure({
"io_worker_count": 8,
"buffer_size": 131072, # 128KB buffer
"close_timeout_ms": 2000,
})
# Reset to defaults
ayafileio.reset_config()| Option | Default | Description |
|---|---|---|
handle_pool_max_per_key |
64 | Max cached handles per file (Windows) |
handle_pool_max_total |
2048 | Max total cached handles (Windows) |
io_worker_count |
0 | IO worker threads, 0=auto |
buffer_pool_max |
512 | Max cached buffers |
buffer_size |
65536 | Buffer size in bytes |
close_timeout_ms |
4000 | Close timeout for pending I/O (ms) |
iocp_batch_size |
64 | IOCP batch completion harvest size (Windows, 1–256) |
io_uring_queue_depth |
256 | io_uring queue depth (Linux) |
io_uring_sqpoll |
False | Enable SQPOLL mode (Linux) |
class AsyncFile(Generic[T]):
def __init__(
self, path: str | Path, mode: str = "rb",
encoding: str | None = None,
newline: str | None = None,
errors: str | None = None,
auto_flush: bool = False
): ...
# 读取
async def read(self, size: int = -1) -> T: ...
async def readline() -> T: ...
async def readlines(hint: int = -1) -> list[T]: ...
async def readall() -> T: ... # read(-1) 别名
async def readinto(buf: bytearray | memoryview) -> int: ... # 零拷贝 [仅二进制]
async def chunk(chunk_size: int, *, buf: bytearray | memoryview | None = None) -> AsyncGenerator[memoryview, None]: ... # 流式分块 [仅二进制]
async def read_at(offset: int, size: int = -1) -> bytes: ... # 位置读（pread 语义），不动文件位置 [仅二进制]
async def read_many(spans: Iterable[tuple[int, int]]) -> list[bytes]: ... # 批量位置读，单事件循环周期提交 [仅二进制]
# 写入
async def write(self, data: str | bytes) -> int: ...
async def writelines(lines) -> None: ... # 批量写入
# 位置
async def seek(self, offset: int, whence: int = 0) -> int: ...
async def tell() -> int: ...
async def truncate(size: int) -> None: ...
# 控制
async def flush(self) -> None: ...
async def close(self) -> None: ...
# 属性
@property
def closed(self) -> bool: ...
@property
def name(self) -> str: ...
@property
def mode(self) -> str: ...
# 状态
def readable() -> bool: ...
def writable() -> bool: ...
def seekable() -> bool: ...
def fileno() -> int: ...
def isatty() -> bool: ...
# 迭代器
def __aiter__(self) -> AsyncFile[T]: ...
async def __anext__(self) -> T: ...| Mode | Description |
|---|---|
"r", "rb" |
Read (text/binary) |
"w", "wb" |
Write (text/binary) |
"a", "ab" |
Append (text/binary) |
"x", "xb" |
Exclusive create (text/binary) |
+ added |
Read/write combinations |
def configure(options: dict) -> None: ... # Unified configuration
def get_config() -> dict: ... # Get current configuration
def reset_config() -> None: ... # Reset to defaults
def get_backend_info() -> dict: ... # Get backend informationdef wrap_file(fd: int, mode: str = "rb", *, owns_fd: bool = False) -> AsyncFile[bytes]: ...Wrap an existing file descriptor (int) or a file-like object with fileno() as an AsyncFile, backed by the optimal platform backend. Binary mode only.
def drain_handle_pool() -> None: ... # Drain all cached file handles
def drain_buffer_pool() -> None: ... # Drain all cached I/O buffersUse drain_handle_pool() / drain_buffer_pool() to release pooled resources at runtime — useful after bulk tempfile operations or between benchmark rounds.
Simulating Crawlee's Dataset append pattern — 5,000 records, 50 concurrent writers, each writing a single line and closing the file:
| Platform | ayafileio | aiofiles | Speedup |
|---|---|---|---|
| Windows (NVMe SSD) | 41,336 items/s | 9,658 items/s | 4.28x |
| Linux (NVMe SSD) | 17,688 items/s | 11,455 items/s | 1.54x |
| macOS (NVMe SSD) | 29,837 items/s | 25,522 items/s | 1.17x |
| Windows (6yr old HDD) | 20,251 items/s | 13,011 items/s | 1.56x |
Key findings:
- On Windows enterprise SSD, ayafileio achieves 42x lower P99 latency (0.044ms vs 1.854ms)
- aiofiles shows 96.7% jitter under load; ayafileio only 16.2%
- Even on degraded hardware, ayafileio maintains predictable performance
Test environment: Windows 10/11, Ubuntu 22.04, macOS 14; GitHub Actions enterprise NVMe SSD
The true test of async I/O — 100,000 concurrent tasks performing random 256B reads on a single shared file handle. No open/close overhead, pure I/O path comparison:
| Library | 1K concur | 10K concur | 50K concur | 100K concur |
|---|---|---|---|---|
| ayafileio (IOCP) | 7,487 ops/s | 46,616 ops/s | 28,165 ops/s | 19,290 ops/s |
| aiofiles (threadpool) | 7,706 ops/s | 7,320 ops/s | 2,131 ops/s | 2,130 ops/s |
| sync threadpool | 9,492 ops/s | 9,469 ops/s | 8,840 ops/s | 8,660 ops/s |
| ayafileio vs aiofiles | 1.0x | 6.4x | 13.2x | 9.1x |
Key findings:
- At low concurrency (1K), all approaches are similar — IOCP setup overhead is amortized
- At 10K+ concurrency, aiofiles' thread pool saturates; throughput drops as concurrency increases — from 7,706 down to 2,130 ops/s (72% degradation)
- ayafileio with IOCP gains throughput at 10K (46,616 ops/s) due to batched completion harvesting via
GetQueuedCompletionStatusEx - At 100K concurrency, ayafileio is 9.1x faster than aiofiles on the same HDD
- The synchronous threadpool (mimicking aiofiles' approach) flatlines at ~8,800 ops/s regardless of concurrency — thread contention ceiling
Test environment: Windows 10, Python 3.14.5, WDC WD10EZEX 7200RPM HDD, 20MB file, 256B random reads
500,000 asyncio tasks all reading from a single file via IOCP — testing the library's absolute concurrency ceiling:
| Metric | Value |
|---|---|
| Concurrent tasks | 500,000 |
| Completion time | 21.6s |
| Throughput | 23,116 ops/s |
| Peak memory (RSS) | ~583 MB |
| Errors | 0 |
| Exceptions | 0 |
ayafileio handles half a million concurrent IOCP reads on a single file handle with zero errors. The dual-IOCP worker architecture (2 threads total) processes all 500K completions while aiofiles would require thousands of threads for the same workload — and still be slower.
We tested 14 different configuration combinations (iocp_batch_size, buffer_size, buffer_pool_max, io_worker_count) on the HDD at 100K concurrency. Result: every configuration scored within ±3% of the default. The library's auto-tuned defaults already saturate the disk's physical I/O limit — there is no software bottleneck left to tune.
For NVMe SSDs with >500K IOPS capability, increasing iocp_batch_size to 128–256 and buffer_size to 128KB may yield additional gains. Use ayafileio.configure() to experiment:
ayafileio.configure({
"iocp_batch_size": 128,
"buffer_size": 131072,
})Contributions are welcome! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Add tests
- Ensure benchmarks pass
- Open a pull request
MIT License — see LICENSE for details.
"Slow is a crime, right?"
— Aya Shameimaru, editor-in-chief of Bunbunmaru News