[cpp.replace.general] p14: Clarification of directive-like preprocessing tokens in macro arguments

[BlowingWind314]

Full name of submitter (unless configured in github; will be published with the issue): Yin Xinyu

Reference (section label): [cpp.replace.general]

Issue description:

[cpp.replace.general]/14 says:

If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.

The intention is possibly avoiding multi-line macro invocations that cross text lines and preprocessing directives ([cpp.pre]). Consider:

// ill-formed
#define FOO(...)
FOO(                // text-line
#include <iostream> // pp-directive
)                   // text-line

Although it's impossible to perform a multi-line macro invocation starting at a preprocessing directive, the current wording would accept the following

// well-formed
#define FOO(...)
FOO(                // text-line
#bluh ) bluh        // pp-directive

since the conditionally-supported directive #bluh ) bluh is not within the list of arguments consisting of merely #bluh.

Implementations have no consensus on how to deal with this case (godbolt). By the way they have not even diagnosed the first example as ill-formed.

This can result in an even weirder situation, although results are all ill-formed. Consider:

#define FOO(...)
#define LPAREN (
#define NIL
FOO(                // text-line
#if LPAREN X ) NIL  // pp-directive
#endif              // pp-directive

and

#define FOO(...)
#ifdef X
FOO(                // text-line
#else ))            // pp-directive
#endif              // pp-directive

It's hard to tell what would happen, where X may be in different branches.

To avoid these complex and anti-intuitive circumstances, we'd better edit the wording of [cpp.replace.general]/14 to forbid these cases.

If an application such as stringization is needed, the following is well-formed.

#define STR(x) STR2(x)
#define STR2(x) #x
#define HASH #
#define NIL
STR(
    HASH include <iostream>
)
STR(
    NIL #include <iostream>
)

The arguments in STR2 is safe, considering [cpp.rescan]/4

The resulting completely macro-replaced preprocessing token sequence is not processed as a preprocessing directive even if it resembles one, but all pragma unary operator expressions within it are then processed as specified in [cpp.pragma.op] below.

Suggested resolution:

Change [cpp.replace.general]/14:

If there are sequences of preprocessing tokens within the list of arguments that , along with all preprocessing tokens immediately following them, would otherwise act as preprocessing directives, the program is ill-formed.

[jensmaurer]

What's the behavior of existing implementations for these examples?

[jensmaurer]

In any case, I don't think those two rules conflict, but they are complementary. [cpp.replace.general] p14 is talking about the argument list (before any replacement happens). [cpp.rescan] p4 is talking about the result of the replacement and the rescanning. At that stage, preprocessing directives are no longer recognized (and the token sequence survives until phase 6, where they are ill-formed, because # is not a valid phase 7 token).

[BlowingWind314]

In any case, I don't think those two rules conflict, but they are complementary. [cpp.replace.general] p14 is talking about the argument list (before any replacement happens). [cpp.rescan] p4 is talking about the result of the replacement and the rescanning. At that stage, preprocessing directives are no longer recognized (and the token sequence survives until phase 6, where they are ill-formed, because # is not a valid phase 7 token).

That's exactly my viewpoint. However even though the first ECHO example would be ill-formed afterward, the second STR example works.

What's the behavior of existing implementations for these examples?

See godbolt. It seems that their preprocessors remain to respect the old specification ([cpp.replace.general]/2 used to be a UB) so they accept both STR(#py comment) and STR(PY comment).

Edit: adding C++ version flag (-std=c++26 etc.) doesn't result in any change.

[jensmaurer]

the old specification ([cpp.replace.general]/2 used to be a UB)

Ah, we changed this in "P2843R3 Preprocessing is never undefined". My guess is implementations haven't caught up to this change. Feel free to post bug reports for

STR(#py comment)  // ill-formed

[BlowingWind314]

the old specification ([cpp.replace.general]/2 used to be a UB)

Ah, we changed this in "P2843R3 Preprocessing is never undefined". My guess is implementations haven't caught up to this change. Feel free to post bug reports for

STR(#py comment)  // ill-formed

That's why I post this issue: is STR(PY comment) well-formed after P2843R2 instead of being an edge case of a UB? In the past, whether STR(PY comment) is well-formed depends on a UB so it doesn't matter at all.

[jensmaurer]

P2843 wanted to remove UB from the preprocessor.

Given the current state of the wording in the standard, is there any reason to believe that STR(PY comment) would be ill-formed?

[BlowingWind314]

P2843 wanted to remove UB from the preprocessor.

Given the current state of the wording in the standard, is there any reason to believe that STR(PY comment) would be ill-formed?

I think STR(PY comment) is well-formed according to the current standard. I wonder if this case needs an informative note or example, since it used to associate with a UB and now the UB is removed but this case becomes well-formed.

[jensmaurer]

Why was it UB? [cpp.replace.general]/14 used to say "UB" (instead of ill-formed), but we agreed this rule is not triggered by your example.

[BlowingWind314]

I found another ambiguity here.

If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.

Does it mean that there shall be no # within the list of arguments since a single # suffices to form a null directive?

Or we should consult [cpp.pre]/2 and whitespaces matter?

A preprocessing directive consists of a sequence of preprocessing tokens that satisfies the following constraints: At the start of translation phase 4, the first preprocessing token in the sequence, referred to as a directive-introducing token , begins with the first character in the source file (optionally after whitespace containing no new-line characters) or follows whitespace containing at least one new-line character, and is

• a # preprocessing token, or
• ...
  The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.

Consider:

STR(
#
) // ill-formed
STR(#) // ill-formed?
STR(#
) // ill-formed?
STR(
#) // ill-formed?

STR(bluh
#
bluh) // ill-formed?
STR(bluh#) // ill-formed?
STR(bluh#if) // ill-formed?  (triggered by #, not by incomplete #if)
STR(bluh#
bluh) // ill-formed?
STR(bluh
#if) // ill-formed?

Should I open a new issue? Or may I edit the current issue to update this problem?

[BlowingWind314]

Why was it UB? [cpp.replace.general]/14 used to say "UB" (instead of ill-formed), but we agreed this rule is not triggered by your example.

Yes if we adopt my interpretation, then STR(PY comment) doesn't belong to the UB.

[Halalaluyafail3]

I found another ambiguity here.

If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.

Does it mean that there shall be no # within the list of arguments since a single # suffices to form a null directive?

Or we should consult [cpp.pre]/2 and whitespaces matter?

Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the # on the start of a line are invalid.

[BlowingWind314]

Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the # on the start of a line are invalid.

How about the ending condition?

The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.

Do you mean that only

STR(bluh
#
bluh)

is ill-formed? Or

STR(bluh
#if)

is also ill-formed? Note that here #if is imcomplete and cannot act as a preprocessing directive otherwise.

[Halalaluyafail3]

Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the # on the start of a line are invalid.

How about the ending condition?

The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.

Do you mean that only

STR(bluh
#
bluh)
is ill-formed? Or

STR(bluh
#if)
is also ill-formed? Note that here #if is imcomplete and cannot act as a preprocessing directive otherwise.

With your previous example:

STR(
#
) // ill-formed
STR(#) // valid
STR(#
) // valid
STR(
#) // ill-formed

STR(bluh
#
bluh) // ill-formed
STR(bluh#) // valid
STR(bluh#if) // valid
STR(bluh#
bluh) // valid
STR(bluh
#if) // ill-formed

The #if) should still be considered a preprocessor directive, just an invalid one. Note that it cannot be treated as a normal text line "A sequence of preprocessing tokens is only a text-line if it does not begin with a directive-introducing token." [cpp.pre]/3. Since it is not a valid if-group, it appears that the whole file is simply not a valid preprocessing-file like the example after the quoted text.

As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.

[BlowingWind314]

As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.

Well [cpp.pre]/2 actually specifies two constraints on the "sequence of preprocessing tokens" that to be considered as a preprocessing directive:

1. The first pp token shall be at the start of a line (optionally after some whitespace containing no new-line).
2. The last pp token shall be immediately followed by some whitespace containing a new-line.

In my opinion, the second constraint is somewhat more important since otherwise there are two possible "sequences of preprocessing tokens" in

STR(
#bluh)

that would otherwise act as preprocessing directives (one is the null directive # and another is the conditionally-supported directive #bluh). Afterall, there is no constraint on when to stop if the second constraint is omiited.

My intention to use imcomplete #if is to distinguish whether the former case (# itself with immediately following pp tokens) is ill-formed. If the if-section causes complexity, one can use

STR(
#line) // is the # itself forms a sequence of pp tokens that would otherwise act as pp-directive?

instead to prohibit the latter case (#line needs more pp tokens) as a valid preprocessing directive otherwise. Don't you agree that "act as" may need the pp-directive to be valid? If we adopt this interpretation of "act as pp-directive", then # itself would make it ill-formed even though #line would not. However this divergence may result in no difference: nevertheless it would be ill-formed because of the appearance of# (my opinion) or because of the invalid but sufficient appearance of #line (your opinion).

[Halalaluyafail3]

As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.

Well [cpp.pre]/2 actually specifies two constraints on the "sequence of preprocessing tokens" that to be considered as a preprocessing directive:

1. The first pp token shall be at the start of a line (optionally after some whitespace containing no new-line).
2. The last pp token shall be immediately followed by some whitespace containing a new-line.

In my opinion, the second constraint is somewhat more important since otherwise there are two possible "sequences of preprocessing tokens" in

STR(
#bluh)
that would otherwise act as preprocessing directives (one is the null directive # and another is the conditionally-supported directive #bluh). Afterall, there is no constraint on when to stop if the second constraint is omiited.

Does it matter which directive it is considering? Though I think it should be considering the whole #bluh), perhaps the wording should be adjusted to not imply that the whole directive needs to be in the arguments.

My intention to use imcomplete #if is to distinguish whether the former case (# itself with immediately following pp tokens) is ill-formed. If the if-section causes complexity, one can use

STR(
#line) // is the # itself forms a sequence of pp tokens that would otherwise act as pp-directive?
instead to prohibit the latter case (#line needs more pp tokens) as a valid preprocessing directive otherwise. Don't you agree that "act as" may need the pp-directive to be valid? If we adopt this interpretation of "act as pp-directive", then # itself would make it ill-formed even though #line would not. However this divergence may result in no difference: nevertheless it would be ill-formed because of the appearance of# (my opinion) or because of the invalid but sufficient appearance of #line (your opinion).

I do not think there is any reason to outright forbid # tokens entirely in preprocessing macros, that was surely never the intent. The intent as far as I understand is that if a macro invocation starts in a text line, it shall not cross over into lines other than text lines. For example:

#define FOO(...)
FOO(
    #include<iostream>
)

This is not valid, but FOO(#) has no issues with it being interpreted as a preprocessing directive so it does not make sense to forbid it. For macro invocations that start in lines other than text lines, there is also a constraint that they must be entirely contained within that line.