feat: add base64 support by RatanKokal · Pull Request #1171 · fortran-lang/stdlib

RatanKokal · 2026-03-31T19:25:19Z

Adds support for base64

Solves issue: #916

This PR provides a portable, branchless scalar implementation. A follow-up PR is in development to add an AVX2-accelerated backend for supported x86_64 architectures.

Copilot

Pull request overview

This PR introduces Base64 encoding/decoding support to stdlib, integrating it into the strings component and adding a dedicated test suite (per issue #916).

Changes:

Added new stdlib_base64 module implementing base64_encode (generic over numeric/logical types) and base64_decode.
Re-exported base64_encode/base64_decode via stdlib_strings.
Added Base64 unit tests and wired them into the string test CMake configuration.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`src/strings/stdlib_base64.fypp`	New Base64 implementation (encode/decode).
`src/strings/stdlib_strings.fypp`	Re-exports Base64 APIs from `stdlib_base64`.
`src/strings/CMakeLists.txt`	Adds `stdlib_base64.fypp` to strings build sources.
`test/string/test_base64.f90`	Adds unit tests for known vectors, whitespace handling, invalid input, and roundtrips.
`test/string/CMakeLists.txt`	Registers the new `base64` test target.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

RatanKokal · 2026-04-04T22:19:25Z

@jvdp1 Could you please have a look? I have tackled all possible issues.

Working on a performance improvement using avx2 specific instructions which was the ultimate goal of the resources shared in the issue. Upon completion, would increase speeds drastically.

Since fortran is depndent on autovectorizers, it may not always lead to the optimall register management as can be controlled using C intrinsics.

Will still have a architecture fallback. Architecture specific kernels can be added to increase performance in future.

jvdp1

thank you @RatanKokal for this PR. Overall it looks already very good to me.

Maybe some general comments:

I suggest to use state_type (provided by stdlib_error to handle errors
I think that some procedures could be declared pure
Is there any reason to prefer module instead of using submodule?

codecov · 2026-04-05T08:20:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.20%. Comparing base (72698fd) to head (7000f3a).
⚠️ Report is 71 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1171      +/-   ##
==========================================
- Coverage   68.66%   68.20%   -0.46%     
==========================================
  Files         408       19     -389     
  Lines       13619     2378   -11241     
  Branches     1537        0    -1537     
==========================================
- Hits         9351     1622    -7729     
+ Misses       4268      756    -3512

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: RatanKokal <ratanskokal@gmail.com>

…ulator Replaced O(N) string searches with an instant Look-Up Table and implemented a 32-bit accumulator to process data in a single pass. Eliminated multi-pass string packing, removed helper function overhead (dead code elimination), and pre-allocated memory to prevent expensive reallocations. Signed-off-by: RatanKokal <ratanskokal@gmail.com>

This commit addresses reviewer feedback regarding memory safety, type portability, and documentation for the pure Fortran implementation. Key changes: - API Alignment: Updated base64_encode_into to mirror the decoder API by returning encoded_len and error_flag. - Memory Safety: Added explicit blank-filling and strict bounds checking to the encoder to prevent silent buffer overflows. - Branchless Bounds Checking: Implemented an iand(..., 255_int32) mask in the decoder to safely handle non-ASCII codepoints. - Strict Portability: Enforced int32 type consistency across all bitwise intrinsics and literal masks. - Documentation: Added FORD headers to the umbrella module. Signed-off-by: RatanKokal <ratanskokal@gmail.com>

This commit hardens the base64 implementation by addressing standard compliance issues and optimizing the hot paths for the zero-copy API. Key changes: * Compliance: Replaced non-conforming 'c_loc' and 'c_f_pointer' usage with 'transfer' and 'select rank' blocks to ensure compatibility with non-C-interoperable types. * Correctness: Fixed a Unicode aliasing bug in the decoder by implementing a bitwise-OR 'unicode_accum' check to reject non-ASCII characters (>255). * Performance: Optimized 'base64_encode_into' by moving string blank-filling ('str = ""') off the hot path, preventing unnecessary memory operations during high-throughput encoding. * API Refinement: Restricted the 'base64_encode_into' power-user API to accept only 'integer(int8)' arrays to guarantee zero-copy performance while remaining standard-compliant. * Ergonomics: Added an optional 'error_flag' to 'base64_decode' to improve error handling for the allocating API. * Testing: Expanded unit tests to include coverage for the new subroutine signatures and error states. Signed-off-by: RatanKokal <ratanskokal@gmail.com>

This commit addresses the architectural and stylistic feedback for the Base64 implementation: * Architecture: Consolidated interfaces into a single stdlib_base64 parent module and moved the high-performance logic into encode and decode submodules to prevent compilation cascades. * Standards: Replaced the logical error flag with type(state_type) from stdlib_error for consistent library-wide error handling. * Optimization: Marked the core _into subroutines as pure to guarantee no side effects and allow for aggressive compiler optimizations. * Dependency: Imported base64_alphabet directly from stdlib_ascii to reduce redundancy. * Documentation: Added explanatory comments for the branchless DT and IS_VAL lookup tables. * Formatting: Expanded semicolon-separated multi-statement lines into single lines to improve debuggability and prevent CI truncation errors. Signed-off-by: RatanKokal <ratanskokal@gmail.com>

RatanKokal · 2026-04-11T12:31:21Z

Hey @jvdp1, could you please have a look. I have encorporated all the feedback.

jvdp1

Thank you @RatanKokal . this PR is taking a good shape. Below are a few comments. Could you also add the specs, please?

jvdp1 · 2026-04-11T17:27:29Z

+
+#:for k1, t1, _, _ in REAL_KINDS_TYPES
+    module function base64_encode_real_${k1}$(data) result(str)
+        ${t1}$, intent(in), target, contiguous :: data(..)


Why is target mentioned?

I agree, target is not needed here, please remove.
The target attribute in fortran has a very specific use and it is to signal that a variable will be pointed to.

jvdp1 · 2026-04-11T17:29:55Z

+
+#:for k1, t1, _ in CMPLX_KINDS_TYPES
+    module function base64_encode_cmplx_${k1}$(data) result(str)
+        ${t1}$, intent(in), target, contiguous :: data(..)


Suggested change

${t1}$, intent(in), target, contiguous :: data(..)

${t1}$, intent(in), contiguous :: data(..)

target is a compiler hint, ennsuring maximum memory efficiency by forcing pass-by-reference.

Where in the fortran standard does it specify such conclusion?
target has a very specific purpose: signaling the compiler that a variable can be "pointed to" by a pointer variable. Telling the compiler that a variable is pontetially a target, can prevent agressive optimization.

Also, in Fortran, arguments are passed by reference.

Please remove this attribute.

jvdp1 · 2026-04-11T17:33:00Z

+    module function base64_encode_bytes(bytes) result(str)
+        integer(int8), intent(in), target, contiguous :: bytes(:)
+        character(len=:), allocatable, target :: str
+        integer(c_size_t) :: nbytes


Could you add a comment to explain why nbytes must be of type c_size_t?

I think this simply requires integer(int64), there is no need to invoke iso_c_binding here.

I agree with @jalvesz : use integer(int64) instead of c_size_t

- use MAXRANK-based rank generation (including rank-0 scalar support) - align c_bool import with stdlib_kinds and keep iso_c_binding for c_size_t - derive base64 alphabet from letters and digits constants - simplify decode despace loop by skipping filtered-out bytes directly - tidy roundtrip tests: parameterized constants and 132-char-safe line split Signed-off-by: RatanKokal <ratanskokal@gmail.com>

Signed-off-by: RatanKokal <ratanskokal@gmail.com>

RatanKokal · 2026-04-13T09:09:44Z

Hi @jvdp1,
I have implemented most of the changes as per the comments.
Also updated the specs.

jalvesz · 2026-04-15T20:43:15Z

+
+#:for k1, t1, _ in CMPLX_KINDS_TYPES
+    module function base64_encode_cmplx_${k1}$(data) result(str)
+        ${t1}$, intent(in), target, contiguous :: data(..)


Where in the fortran standard does it specify such conclusion?
target has a very specific purpose: signaling the compiler that a variable can be "pointed to" by a pointer variable. Telling the compiler that a variable is pontetially a target, can prevent agressive optimization.

Also, in Fortran, arguments are passed by reference.

Please remove this attribute.

jalvesz · 2026-04-15T20:48:27Z

+    module function base64_encode_bytes(bytes) result(str)
+        integer(int8), intent(in), target, contiguous :: bytes(:)
+        character(len=:), allocatable, target :: str
+        integer(c_size_t) :: nbytes


I think this simply requires integer(int64), there is no need to invoke iso_c_binding here.

- Aligned `fypp` preprocessor macros to match their respective inner scopes. - Removed redundant `target` attributes from dummy arguments. - Vectorized decoding routines using a `chunk_size` parameter and array constructors. - Simplified `iachar` intrinsic calls by utilizing the `kind` argument. - Replaced `c_size_t` with `int64` for byte tracking and added explanatory documentation. - Renamed `base64_alphabet` to `base64_dictionary` for improved clarity. - Converted documentation snippets into executable examples and integrated them into the CMake build system.

RatanKokal · 2026-07-13T15:59:11Z

@jvdp1 @jalvesz I have incorporated the feedback/changes requested. Requesting you to please have a look

jalvesz · 2026-07-15T20:16:34Z

+            slen = len(str)
+            if (mod(slen, 4) /= 0) then
+                err_state = state_type("base64_decode_into", STDLIB_VALUE_ERROR, &
+                    "Input length must be a multiple of 4")


Suggested change

"Input length must be a multiple of 4")

"Input length must be a multiple of "//to_string(chunk_size))

Unfortunately, to_string cannot be used here because base64_decode_into must remain a pure subroutine, and calling an impure function results in a compilation error. To remove the hardcoded number while preserving the procedure's purity, I defined an FYPP macro (#:set ENC_CHUNK = 4) and injected it directly into the string at compile time.

RatanKokal · 2026-07-17T04:59:58Z

@jalvesz I have used enc_chunk = 4 and raw_chunk = 3 now, 3 bytes of original data gets encoded as 4 bytes. This would give a even more clear picture in my opinion. Requesting you to please have a look.

RatanKokal · 2026-07-19T07:15:28Z

@jalvesz I have introduced two variables for chunk size, raw chunk and enc chunk which is 3 and 4 respectively since base64 converts 3 bytes of normal text in 4 bytes of encoded text

jalvesz · 2026-07-19T14:48:33Z

@RatanKokal is there a reason for converting the size parameters to fypp parameters? This looks like an overkill complexity.

RatanKokal · 2026-07-19T15:06:22Z

@RatanKokal is there a reason for converting the size parameters to fypp parameters? This looks like an overkill complexity.

I get error in "Input length must be a multiple of "//to_string(chunk_size)) if not done this way. to_string cannot be used here because base64_decode_into must remain a pure subroutine, and calling an impure function results in a compilation error.

jalvesz · 2026-07-19T16:46:53Z

stdlib/src/strings/stdlib_strings_to_string.fypp

Line 12 in 9a15c77

pure module function to_string_${t1[0]}$_${k1}$(value, format) result(string)

The procedure to_string is pure. I think the error you got was simply that the module containing it was not used

RatanKokal · 2026-07-20T06:01:27Z

@jalvesz Thanks for the correction. Made the changes and it does work now.

jvdp1

thank you @RatanKokal . Overall it looks good to me. Here are some suggestions

jvdp1 · 2026-07-20T06:11:44Z

+`data`: shall be a contiguous array of an intrinsic type (`real`, `integer`,
+`complex`, `logical`, or `integer(int8)`). It is an `intent(in)` argument.


integer is mentioned twice. Does it mean that all kinds of integer are supported? (and what about other types?)
Maybe:

Suggested change

`data`: shall be a contiguous array of an intrinsic type (`real`, `integer`,

`complex`, `logical`, or `integer(int8)`). It is an `intent(in)` argument.

`data`: shall be a contiguous array of an intrinsic type (`real`, `integer`,

`complex`, `logical`) of any kind. It is an `intent(in)` argument.

Updated the docs to make it clear. Yeah everything is supported.

jvdp1 · 2026-07-20T06:12:51Z

+Encodes bytes into a caller-provided output buffer.
+
+This is the preallocated API for throughput-sensitive workflows. It does not
+allocate and reports status through `err_state`. On success,


The sentence is unclear. Does it or not report status?

Made it clear, it reports status

jvdp1 · 2026-07-20T06:13:46Z

+`bytes`: shall be a contiguous array of type `integer(int8)`. It is an
+`intent(in)` argument.
+
+`str`: shall be an intrinsic character type. It is an `intent(out)` argument.


Is it allocatable?

jvdp1 · 2026-07-20T06:21:25Z

+
+module stdlib_base64
+    use stdlib_kinds, only: sp, dp, xdp, qp, int8, int16, int32, int64, lk, c_bool
+    use stdlib_ascii, only: base64_dictionary => base64_alphabet


Why did you rename it? For readability, I suggest to avoid renaming variables, unless it improves readability, or it avoids issues likes name conflicts.

I was requested to have a more explicit name like dictionary, and I wanted to have some resemblance with the original RFQ. I have updated this in my latest commit to use base64_dictionary everywhere.

jvdp1 · 2026-07-20T06:23:38Z

+        !> Encodes bytes into a caller-provided output buffer.
+        !>
+        !> This is the preallocated API for throughput-sensitive workflows.
+        !> It does not allocate and reports status through `err_state`.


Suggested change

!> It does not allocate and reports status through `err_state`.

!> It does not allocate `str` and reports status through `err_state`.

or

Suggested change

!> It does not allocate and reports status through `err_state`.

!> `str` must be of a defined length. It reports status through `err_state`.

Made these lines more clear

jvdp1 · 2026-07-20T06:40:57Z

+            allocate(filtered(len(str)))
+            j = 0
+            do i = 1, len(str)
+                raw_c = iachar(str(i:i), kind=int32)
+                unicode_accum = ior(unicode_accum, raw_c)
+                c = iand(raw_c, 255_int32)
+                if (IS_VAL(c) == 0) cycle
+                j = j + 1
+                filtered(j) = int(c, int8)
+            end do
+            slen = j


some entries of filtered might be undefined due to cycle. I suggest to mention it in the specs.

I have added comment in the code, since filtered is dev facing

jvdp1 · 2026-07-20T06:48:12Z

+                v = int(DT(iand(raw_c4, 255_int32)), int32)
+            else
+                v = int(DT(iand(int(filtered(i:i+enc_chunk-1), int32), 255_int32)), int32)


DT (declared as integer(int8)) seems to be always converted to int32. For readability and efficiency, I suggest to declared DT as integer(int32)

jvdp1 · 2026-07-20T06:54:57Z

+    integer, parameter :: enc_chunk = 4
+    integer, parameter :: raw_chunk = 3


Since these variables are declared in both modules, I suggest to define them only once in the parent module.

Agreed, done

jvdp1 · 2026-07-20T07:00:00Z

+        call check(error, base64_encode([int(77, int8), int(97, int8), int(110, int8)]) == "TWFu")
+        if (allocated(error)) return
+        call check(error, base64_encode([int(77, int8), int(97, int8)]) == "TWE=")
+        if (allocated(error)) return
+        call check(error, base64_encode([int(77, int8)]) == "TQ==")
+        if (allocated(error)) return
+
+        call check(error, base64_decode("TWFu") == "Man")
+        if (allocated(error)) return
+        call check(error, base64_decode("TWE=") == "Ma")
+        if (allocated(error)) return
+        call check(error, base64_decode("TQ==") == "M")


It would be good to add some message in check. It will be helpful to identify which check is failing, in case one of them will fail. Currenty, the developer won't be able to identify it without additional work.

jvdp1 · 2026-07-20T07:00:36Z

+        integer(int32) :: vals(4), got(4)
+        character(len=:), allocatable :: enc, dec
+
+        vals = [1_int32, -2_int32, 1024_int32, -4096_int32]


could it be declared asparameter?

- Rename base64_alphabet to base64_dictionary - Move chunk parameters to parent module - Optimize DT lookup table to int32 - Improve test error messages and docstrings

RatanKokal · 2026-07-20T13:58:50Z

@jvdp1 Thanks for the feedback. I have incorporated all of that in my latest commit. Requesting you to have a look.
Thanks.

RatanKokal · 2026-07-25T07:16:17Z

@jalvesz Are there any more suggestions, or are we good to go with this one?

jvdp1 requested a review from Copilot April 3, 2026 19:07

Copilot started reviewing on behalf of jvdp1 April 3, 2026 19:08 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread src/strings/stdlib_base64.fypp Outdated

Comment thread src/strings/stdlib_base64.fypp Outdated

Comment thread src/strings/stdlib_base64.fypp

RatanKokal requested a review from Copilot April 4, 2026 20:53

Copilot started reviewing on behalf of RatanKokal April 4, 2026 20:54 View session

Copilot AI reviewed Apr 4, 2026

View reviewed changes

RatanKokal force-pushed the wip_base64 branch from 92108e4 to 2e7fad9 Compare April 5, 2026 07:22

jvdp1 requested changes Apr 5, 2026

View reviewed changes

RatanKokal added 5 commits April 5, 2026 15:18

feat: add base64 support

3d575e2

Signed-off-by: RatanKokal <ratanskokal@gmail.com>

RatanKokal force-pushed the wip_base64 branch from 7235672 to e21a325 Compare April 5, 2026 09:53

RatanKokal requested a review from jvdp1 April 5, 2026 09:54

jvdp1 reviewed Apr 11, 2026

View reviewed changes

RatanKokal requested a review from jvdp1 April 13, 2026 08:58

docs(base64): add stdlib_base64 spec

4b9ac9b

Signed-off-by: RatanKokal <ratanskokal@gmail.com>

jalvesz requested changes Apr 16, 2026

View reviewed changes

jvdp1 reviewed Apr 17, 2026

View reviewed changes

Comment thread src/strings/stdlib_base64_encode.fypp Outdated

14NGiestas added the topic: strings String processing label May 7, 2026

14NGiestas linked an issue May 7, 2026 that may be closed by this pull request

base64 #916

Open

RatanKokal requested review from jalvesz and jvdp1 July 13, 2026 05:31

jalvesz requested changes Jul 16, 2026

View reviewed changes

RatanKokal force-pushed the wip_base64 branch from d0b0566 to 81fce42 Compare July 17, 2026 04:55

RatanKokal requested a review from jalvesz July 17, 2026 05:00

RatanKokal force-pushed the wip_base64 branch from 81fce42 to 1db99db Compare July 17, 2026 05:25

RatanKokal force-pushed the wip_base64 branch from 1db99db to 6c0f797 Compare July 19, 2026 12:33

refactor(base64): Use enc_chunk and raw_chunk

1ed95c4

RatanKokal force-pushed the wip_base64 branch from 6c0f797 to 1ed95c4 Compare July 20, 2026 06:00

jvdp1 reviewed Jul 20, 2026

View reviewed changes

refactor(base64): address PR review comments

7000f3a

- Rename base64_alphabet to base64_dictionary - Move chunk parameters to parent module - Optimize DT lookup table to int32 - Improve test error messages and docstrings

RatanKokal requested a review from jvdp1 July 20, 2026 14:01

	${t1}$, intent(in), target, contiguous :: data(..)
	${t1}$, intent(in), contiguous :: data(..)

	"Input length must be a multiple of 4")
	"Input length must be a multiple of "//to_string(chunk_size))

		`data`: shall be a contiguous array of an intrinsic type (`real`, `integer`,
		`complex`, `logical`, or `integer(int8)`). It is an `intent(in)` argument.

	!> It does not allocate and reports status through `err_state`.
	!> It does not allocate `str` and reports status through `err_state`.

	!> It does not allocate and reports status through `err_state`.
	!> `str` must be of a defined length. It reports status through `err_state`.

		integer, parameter :: enc_chunk = 4
		integer, parameter :: raw_chunk = 3

Uh oh!

Conversation

RatanKokal commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RatanKokal commented Apr 4, 2026

Uh oh!

jvdp1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

RatanKokal commented Apr 11, 2026

Uh oh!

jvdp1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RatanKokal commented Apr 13, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RatanKokal commented Mar 31, 2026 •

edited

Loading

codecov Bot commented Apr 5, 2026 •

edited

Loading