Cannot compile helper tools with libc++

[viniciusferrao]

As discussed on #9 it seems that libc++ does not implement the spaceship operator, which breaks the library when libc++ is used instead of libstdc++:

On macOS, it defaults to libc++, and it's impossible to compile the helper tools:

FAILED: merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o 
/opt/local/bin/clang++ -DFMT_LOCALE -DFMT_SHARED -isystem /opt/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -std=c++20 -MD -MT merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o -MF merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o.d -o merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o -c /Users/ferrao/Documents/Development/i18n++/merge/merge.cpp
/Users/ferrao/Documents/Development/i18n++/merge/merge.cpp:15:44: error: invalid operands to binary expression ('const std::optional<std::string>' (aka 'const optional<basic_string<char, char_traits<char>, allocator<char>>>') and 'const std::optional<std::string>')
    std::strong_ordering order = a.context <=> b.context;
                                 ~~~~~~~~~ ^   ~~~~~~~~~

[zauguin]

@viniciusferrao Could you test if the libc++ branch works on Mac?

[viniciusferrao]

The first tries were no good:

❯ cmake --build build                                                                                                                                        
[51/88] Building CXX object merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o
FAILED: merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o 
/opt/local/bin/clang++-mp-13 -DFMT_LOCALE -DFMT_SHARED -isystem /opt/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -std=c++20 -MD -MT merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o -MF merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o.d -o merge/CMakeFiles/i18n-merge-pot.dir/merge.cpp.o -c /Users/ferrao/i18n++/merge/merge.cpp
/Users/ferrao/i18n++/merge/merge.cpp:39:54: error: invalid operands to binary expression ('const std::vector<std::pair<std::optional<std::string>, std::optional<std::string>>>' (aka 'const vector<pair<optional<basic_string<char, char_traits<char>, allocator<char>>>, optional<basic_string<char, char_traits<char>, allocator<char>>>>>') and 'const std::vector<std::pair<std::optional<std::string>, std::optional<std::string>>>')
    std::strong_ordering order = a.extractedComments <=> b.extractedComments;
                                 ~~~~~~~~~~~~~~~~~~~ ^   ~~~~~~~~~~~~~~~~~~~
/opt/local/libexec/llvm-13/bin/../include/c++/v1/__algorithm/sift_down.h:44:34: note: in instantiation of function template specialization '(anonymous namespace)::postsort_msgs(std::vector<Message> &)::(anonymous class)::operator()<client::ast::Message, client::ast::Message>' requested here
    if ((__child + 1) < __len && __comp(*__child_i, *(__child_i + 1))) {
                                 ^
/opt/local/libexec/llvm-13/bin/../include/c++/v1/__algorithm/make_heap.h:38:20: note: in instantiation of function template specialization 'std::__sift_down<(lambda at /Users/ferrao/i18n++/merge/merge.cpp:38:39) &, std::__wrap_iter<client::ast::Message *>>' requested here
            _VSTD::__sift_down<_Compare>(__first, __last, __comp, __n, __first + __start);
                   ^
/opt/local/libexec/llvm-13/bin/../include/c++/v1/__algorithm/partial_sort.h:35:12: note: in instantiation of function template specialization 'std::__make_heap<(lambda at /Users/ferrao/i18n++/merge/merge.cpp:38:39) &, std::__wrap_iter<client::ast::Message *>>' requested here
    _VSTD::__make_heap<_Compare>(__first, __middle, __comp);
           ^
/opt/local/libexec/llvm-13/bin/../include/c++/v1/__algorithm/sort.h:512:16: note: in instantiation of function template specialization 'std::__partial_sort<(lambda at /Users/ferrao/i18n++/merge/merge.cpp:38:39) &, std::__wrap_iter<client::ast::Message *>>' requested here
        _VSTD::__partial_sort<_Comp_ref>(__first, __last, __last, _Comp_ref(__comp));
               ^
/Users/ferrao/i18n++/merge/merge.cpp:38:8: note: in instantiation of function template specialization 'std::sort<std::__wrap_iter<client::ast::Message *>, (lambda at /Users/ferrao/i18n++/merge/merge.cpp:38:39)>' requested here
  std::sort(msgs.begin(), msgs.end(), [](const auto &a, const auto &b) {

I've tried with AppleClang and Clang from Macports. None of them were able to compile.

For AppleClang I've used:
Clang_DIR=/opt/local/libexec/llvm-13 cmake -B build -G Ninja

And for Macports Clang:
CC=/opt/local/bin/clang-mp-13 CXX=/opt/local/bin/clang++-mp-13 Clang_DIR=/opt/local/libexec/llvm-13 cmake -B build -G Ninja

Unfortunately "Clang_DIR" is only available on the Macports release of Clang. I've looked on my filesystem for the usual suspects but it's just missing:

/Library/Developer/CommandLineTools/usr/include/c++/v1/float.h
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/float.h
/Library/Developer/CommandLineTools/usr/lib/tapi/13.0.0/include/float.h
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/float.h
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/float.h
/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/float.h
/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/float.h
/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/Resources/Clang/include/float.h

None of those worked.

I'm attaching both variants, since the output is different, so you can check it better. It's still a libc++ issue. I think when I use Macports Clang it goes for libc++ 13.0 but I'm not 100% sure.

AppleClang.txt
MacportsClang.txt

Thank you.

EDIT: Checked libc++ versions

❯ cat /opt/local/libexec/llvm-13/include/c++/v1/__libcpp_version
13000
❯ cat /Library/Developer/CommandLineTools/usr/include/c++/v1/__libcpp_version
11000
❯ cat /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/__libcpp_version
12000

The question is: now, which one macOS uses by default? 11000 or 12000? At least Macports is with libc++ 13.

[zauguin]

The first tries were no good:

Sorry about that, I was stupid and missed that directly under the changed code there was another instance using the spaceship operator...
Should be fixed now. (There is still one use in the example, but I think it's easier just not to build the example than to change it.)

[viniciusferrao]

Definitely there's some progress.

With AppleClang the error is small during the linking, it seems that -fuse-ld=lld is not supported.

FAILED: clang/i18n-extract 
: && /Library/Developer/CommandLineTools/usr/bin/c++ -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fuse-ld=lld clang/CMakeFiles/clang_common.dir/attr.cpp.o clang/CMakeFiles/clang_common.dir/action.cpp.o clang/CMakeFiles/clang_common.dir/__/common/write_po.cpp.o clang/CMakeFiles/i18n-extract.dir/tool.cpp.o -o clang/i18n-extract  -Wl,-rpath,/opt/local/libexec/llvm-13/lib  /opt/local/libexec/llvm-13/lib/libclang-cpp.dylib  /opt/local/libexec/llvm-13/lib/libLLVM.dylib && :
clang: error: invalid linker name in argument '-fuse-ld=lld'
[81/88] Linking CXX shared module clang/i18n-clang.so
FAILED: clang/i18n-clang.so 
: && /Library/Developer/CommandLineTools/usr/bin/c++  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -bundle -Wl,-headerpad_max_install_names -fuse-ld=lld -o clang/i18n-clang.so clang/CMakeFiles/clang_common.dir/attr.cpp.o clang/CMakeFiles/clang_common.dir/action.cpp.o clang/CMakeFiles/clang_common.dir/__/common/write_po.cpp.o clang/CMakeFiles/plugin.dir/plugin.cpp.o   && :
clang: error: invalid linker name in argument '-fuse-ld=lld'
ninja: build stopped: subcommand failed.

However when using Macports Clang the error is way bigger, it seems to be using ld64.lld but it fails.

FAILED: clang/i18n-clang.so 
: && /opt/local/bin/clang++-mp-13  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -bundle -Wl,-headerpad_max_install_names -fuse-ld=lld -o clang/i18n-clang.so clang/CMakeFiles/clang_common.dir/attr.cpp.o clang/CMakeFiles/clang_common.dir/action.cpp.o clang/CMakeFiles/clang_common.dir/__/common/write_po.cpp.o clang/CMakeFiles/plugin.dir/plugin.cpp.o   && :
ld64.lld: error: undefined symbol: llvm::Registry<clang::ParsedAttrInfo>::add_node(llvm::Registry<clang::ParsedAttrInfo>::node*)
>>> referenced by clang/CMakeFiles/clang_common.dir/attr.cpp.o

For the Macports release I'm attaching the full log on a text file, since it's too big.

linkerErrorMacportsClang.txt

[zauguin]

Definitely there's some progress.

That's a start...

With AppleClang the error is small during the linking, it seems that -fuse-ld=lld is not supported.

I think I added that just because the default linker was incredibly slow, it's probably safe to remove it.

That will just trigger the Macports issue though. The problem is that the plugin uses LLVM and clang but does not link against them. That's fine on Linux since it's a shared library and a shared library can have undefined symbols as long as these symbols are defined by the program loading the shared library. It seems like Mac wants all symbols to be defined directly.

Now there are two possible situations: If the clang compiler on your system links to a dynamic libclang library (and probably a libLLVM library) than we could link the plugin against the same library and it should work. But if clang is linked statically than this is more complicated since the plugin can't link to libclang since otherwise we would end up with two independent copies of clang after loading the plugin.

I pushed a change which just instructs the Mac linker to act more like Linux here and therefore hopefully fixes this, but I'm not sure that this will work.

I also added a CMake option I18NPP_BUILD_EXAMPLE which you can set to OFF to avoid compilation errors due to the example using <=>.

[viniciusferrao]

Hi @zauguin, sorry for the delayed answer, I got stuck with a huge work emergency, that seems to be "solved" now. I would like to dig a little further before answering you but since there's 3 days without answer I just ran again with the new patches and the result wasn't good again.

For AppleClang it's fails on build now.

❯ cmake --build build
[82/85] Building CXX object tests/CMakeFiles/tests.dir/simple.cpp.o
FAILED: tests/CMakeFiles/tests.dir/simple.cpp.o 
/Library/Developer/CommandLineTools/usr/bin/c++ -DFMT_LOCALE -DFMT_SHARED -DTEST_SOURCE_DIR=\"/Users/ferrao/i18n++/tests\" -I/Users/ferrao/i18n++/tests/../include -I/Users/ferrao/i18n++/clang/../include -I/Users/ferrao/i18n++/external/Catch2/src/catch2/.. -isystem /opt/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Xclang -plugin-arg-i18n -Xclang nodomain -Xclang -plugin-arg-i18n -Xclang basepath=/Users/ferrao/i18n++ -Xclang -plugin-arg-i18n -Xclang comment=L10N: -fplugin=/Users/ferrao/i18n++/build/clang/i18n-clang.so -std=c++2a -MD -MT tests/CMakeFiles/tests.dir/simple.cpp.o -MF tests/CMakeFiles/tests.dir/simple.cpp.o.d -o tests/CMakeFiles/tests.dir/simple.cpp.o -c /Users/ferrao/i18n++/tests/simple.cpp
error: unable to load plugin '/Users/ferrao/i18n++/build/clang/i18n-clang.so': 'dlopen(/Users/ferrao/i18n++/build/clang/i18n-clang.so, 0x0009): symbol not found in flat namespace '__ZN4llvm25llvm_unreachable_internalEPKcS1_j''
[83/85] Building CXX object tests/CMakeFiles/tests.dir/readme.cpp.o
FAILED: tests/CMakeFiles/tests.dir/readme.cpp.o 
/Library/Developer/CommandLineTools/usr/bin/c++ -DFMT_LOCALE -DFMT_SHARED -DTEST_SOURCE_DIR=\"/Users/ferrao/i18n++/tests\" -I/Users/ferrao/i18n++/tests/../include -I/Users/ferrao/i18n++/clang/../include -I/Users/ferrao/i18n++/external/Catch2/src/catch2/.. -isystem /opt/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Xclang -plugin-arg-i18n -Xclang nodomain -Xclang -plugin-arg-i18n -Xclang basepath=/Users/ferrao/i18n++ -Xclang -plugin-arg-i18n -Xclang comment=L10N: -fplugin=/Users/ferrao/i18n++/build/clang/i18n-clang.so -std=c++2a -MD -MT tests/CMakeFiles/tests.dir/readme.cpp.o -MF tests/CMakeFiles/tests.dir/readme.cpp.o.d -o tests/CMakeFiles/tests.dir/readme.cpp.o -c /Users/ferrao/i18n++/tests/readme.cpp
error: unable to load plugin '/Users/ferrao/i18n++/build/clang/i18n-clang.so': 'dlopen(/Users/ferrao/i18n++/build/clang/i18n-clang.so, 0x0009): symbol not found in flat namespace '__ZN4llvm25llvm_unreachable_internalEPKcS1_j''
ninja: build stopped: subcommand failed.

When using Macports clang the behavior is different, but still not working and with a lot of errors.
MacportsClang.txt

For both tests I've disabled the I18NPP_BUILD_EXAMPLE flag as you recommended. Tomorrow I'll try to dig a little further.

Thanks again.

[zauguin]

Hi,

the Apple Clang issue is caused by compiling against another clang version, so it expects symbols from another version. To fix this the plugin has to be build against the headers of Apple clang, but I don't know if they are installed by default.

Let's focus on the other issue first. I'll deal with the warnings at the beginning later, if you don't want to extract strings from files using the classical gettext functions they are less important. The std::numbers issue should be fixed with the latest commit.

[viniciusferrao]

Hi @zauguin we're almost there, with Macports Clang now there's only a linker error (ignoring the warnings).

[85/85] Linking CXX executable tests/tests
FAILED: tests/tests tests/tests_tests-b12d07c.cmake /Users/ferrao/i18n++/build/tests/tests_tests-b12d07c.cmake 
: && /opt/local/bin/clang++-mp-13 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names  tests/CMakeFiles/tests.dir/simple.cpp.o tests/CMakeFiles/tests.dir/readme.cpp.o -o tests/tests  /opt/local/lib/libfmt.8.1.1.dylib  external/Catch2/src/libCatch2Main.a  external/Catch2/src/libCatch2.a && cd /Users/ferrao/i18n++/build/tests && /opt/local/bin/cmake -D TEST_TARGET=tests -D TEST_EXECUTABLE=/Users/ferrao/i18n++/build/tests/tests -D TEST_EXECUTOR= -D TEST_WORKING_DIR=/Users/ferrao/i18n++/build/tests -D TEST_SPEC= -D TEST_EXTRA_ARGS= -D TEST_PROPERTIES= -D TEST_PREFIX= -D TEST_SUFFIX= -D TEST_LIST=tests_TESTS -D TEST_REPORTER= -D TEST_OUTPUT_DIR= -D TEST_OUTPUT_PREFIX= -D TEST_OUTPUT_SUFFIX= -D CTEST_FILE=/Users/ferrao/i18n++/build/tests/tests_tests-b12d07c.cmake -P /Users/ferrao/i18n++/external/Catch2/extras/CatchAddTests.cmake
Undefined symbols for architecture x86_64:
  "_libintl_bindtextdomain", referenced from:
      bindtextdomain(char const*, char const*) in simple.cpp.o
  "_libintl_dgettext", referenced from:
      dgettext(char const*, char const*) in simple.cpp.o
      dgettext(char const*, char const*) in readme.cpp.o
  "_libintl_dngettext", referenced from:
      dngettext(char const*, char const*, char const*, unsigned long) in simple.cpp.o
      dngettext(char const*, char const*, char const*, unsigned long) in readme.cpp.o
  "_libintl_textdomain", referenced from:
      textdomain(char const*) in simple.cpp.o
      textdomain(char const*) in readme.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

I'll try to find the right headers for AppleClang, if they exist.

I really appreciate the effort. Thanks.

[zauguin]

Hi @zauguin we're almost there, with Macports Clang now there's only a linker error (ignoring the warnings).

@viniciusferrao Hopefully this is now fixed too.

[zauguin]

Let's focus on the other issue first. I'll deal with the warnings at the beginning later

I also tried to do something about the warnings, but there's a fair chance that it broke more than it fixed... Sadly I can't easily test with a freestanding libintl for the moment, so I'll have to rely on you to test it again.

[viniciusferrao]

I think the warning fix made a stepback. With Macports Clang that's happening now:

[83/85] Building CXX object tests/CMakeFiles/tests.dir/simple.cpp.o
FAILED: tests/CMakeFiles/tests.dir/simple.cpp.o 
/opt/local/bin/clang++-mp-13 -DFMT_LOCALE -DFMT_SHARED -DTEST_SOURCE_DIR=\"/Users/ferrao/i18n++/tests\" -I/Users/ferrao/i18n++/tests/../include -I/Users/ferrao/i18n++/clang/../include -I/Users/ferrao/i18n++/external/Catch2/src/catch2/.. -isystem /opt/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Xclang -plugin-arg-i18n -Xclang nodomain -Xclang -plugin-arg-i18n -Xclang basepath=/Users/ferrao/i18n++ -Xclang -plugin-arg-i18n -Xclang comment=L10N: -fplugin=/Users/ferrao/i18n++/build/clang/i18n-clang.so -std=c++20 -MD -MT tests/CMakeFiles/tests.dir/simple.cpp.o -MF tests/CMakeFiles/tests.dir/simple.cpp.o.d -o tests/CMakeFiles/tests.dir/simple.cpp.o -c /Users/ferrao/i18n++/tests/simple.cpp
In file included from /Users/ferrao/i18n++/tests/simple.cpp:2:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:15:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/../i18n.hpp:9:
/opt/local/include/libintl.h:128:7: error: static declaration of 'gettext' follows non-static declaration
char *gettext (const char *__msgid)
      ^
/Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:5:28: note: previous declaration is here
[[mfk::i18n]] extern char *gettext([[mfk::i18n_singular_begin]] const char *) noexcept;
                           ^
In file included from /Users/ferrao/i18n++/tests/simple.cpp:2:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:15:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/../i18n.hpp:9:
/opt/local/include/libintl.h:148:7: error: static declaration of 'dgettext' follows non-static declaration
char *dgettext (const char *__domainname, const char *__msgid)
      ^
/Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:6:28: note: previous declaration is here
[[mfk::i18n]] extern char *dgettext([[mfk::i18n_domain_begin]] const char *,
                           ^
In file included from /Users/ferrao/i18n++/tests/simple.cpp:2:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:15:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/../i18n.hpp:9:
/opt/local/include/libintl.h:192:7: error: static declaration of 'ngettext' follows non-static declaration
char *ngettext (const char *__msgid1, const char *__msgid2,
      ^
/Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:8:28: note: previous declaration is here
[[mfk::i18n]] extern char *ngettext([[mfk::i18n_singular_begin]] const char *,
                           ^
In file included from /Users/ferrao/i18n++/tests/simple.cpp:2:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:15:
In file included from /Users/ferrao/i18n++/tests/../include/i18n/../i18n.hpp:9:
/opt/local/include/libintl.h:215:7: error: static declaration of 'dngettext' follows non-static declaration
char *dngettext (const char *__domainname, const char *__msgid1,
      ^
/Users/ferrao/i18n++/tests/../include/i18n/simple.hpp:10:28: note: previous declaration is here
[[mfk::i18n]] extern char *dngettext([[mfk::i18n_domain_begin]] const char *,
                           ^
4 errors generated.
ninja: build stopped: subcommand failed.

[zauguin]

It's reverted now... Probably the only way to solve this currently is to compile with -D_INTL_REDIRECT_MACROS.

[viniciusferrao]

Alright it seems to be passing now:

❯ ./i18n-extract --version
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: x86_64-apple-darwin21.3.0
  Host CPU: skylake

Now I need to learn how to use the tool. Warnings still applies but effectively is that an issue?

Thanks.