Skip to content

Commit 2a1f8fa

Browse files
lemirelemire
authored
Provides support for clang under Windows. (simdjson#817)
1 parent 04e47bd commit 2a1f8fa

17 files changed

Lines changed: 438 additions & 194 deletions

‎.appveyor.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ environment:
88
matrix:
99
- job_name: VS2019
1010
CMAKE_ARGS: -DSIMDJSON_CHECKPERF_BRANCH=jkeiser/parse-t
11+
- job_name: VS2019CLANG
12+
CMAKE_ARGS: -T ClangCL
1113
- job_name: VS2017 (Static, No Threads)
1214
image: Visual Studio 2017
1315
CMAKE_ARGS: -DSIMDJSON_BUILD_STATIC=ON -DSIMDJSON_ENABLE_THREADS=OFF
@@ -30,5 +32,6 @@ test_script:
3032
- ctest --output-on-failure -C %Configuration% --verbose %CTEST_ARGS% --parallel
3133

3234
clone_folder: c:\projects\simdjson
35+
3336
matrix:
3437
fast_finish: true

‎HACKING.md‎

Lines changed: 67 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,53 @@ While simdjson distributes just two files from the singleheader/ directory, we *
6464
multiple files under include/ and src/. include/simdjson.h and src/simdjson.cpp are the "spine" for
6565
these, and you can include
6666
67+
68+
69+
Runtime Dispatching
70+
--------------------
71+
72+
A key feature of simdjson is the ability to compile different processing kernels, optimized for specific instruction sets, and to select
73+
the most appropriate kernel at runtime. This ensures that users get the very best performance while still enabling simdjson to run everywhere.
74+
This technique is frequently called runtime dispatching. The simdjson achieves runtime dispatching entirely in C++: we do not assume
75+
that the user is building the code using CMake, for example.
76+
77+
To make runtime dispatching work, it is critical that the code be compiled for the lowest supported processor. In particular, you should
78+
not use flags such as -mavx2, /arch:AVX2 and so forth while compiling simdjson. When you do so, you allow the compiler to use advanced
79+
instructions. In turn, these advanced instructions present in the code may cause a runtime failure if the runtime processor does not
80+
support them. Even a simple loop, compiled with these flags, might generate binary code that only run on advanced processors.
81+
82+
So we compile simdjson for a generic processor. Our users should do the same if they want simdjson's runtime dispatch to work. It is important
83+
to understand that if runtime dispatching does not work, then simdjson will cause crashes on older processors. Of course, if a user chooses
84+
to compile their code for a specific instruction set (e.g., AVX2), they are responsible for the failures if they later run their code
85+
on a processor that does not support AVX2. Yet, if we were to entice these users to do so, we would share the blame: thus we carefully instruct
86+
users to compile their code in a generic way without doing anything to enable advanced instructions.
87+
88+
89+
We only use runtime dispatching on x64 (AMD/Intel) platforms, at the moment. On ARM processors, we would need a standard way to query, at runtime,
90+
the processor for its supported features. We do not know how to do so on ARM systems in general. Thankfully it is not yet a concern: 64-bit ARM
91+
processors are fairly uniform as far as the instruction sets they support.
92+
93+
94+
In all cases, simdjson uses advanced instructions by relying on "intrinsic functions": we do not write assembly code. The intrinsic functions
95+
are special functions that the compiler might recognize and translate into fast code. To make runtime dispatching work, we rely on the fact that
96+
the header providing these instructions
97+
(intrin.h under Visual Studio, x86intrin.h elsewhere) defines all of the intrinsic functions, including those that are not supported
98+
processor.
99+
100+
At this point, we are require to use one of two main strategies.
101+
102+
1. On POSIX systems, the main compilers (LLVM clang, GNU gcc) allow us to use any intrinsic function after including the header, but they fail to inline the resulting instruction if the target processor does not support them. Because we compile for a generic processor, we would not be able to use most intrinsic functions. Thankfully, more recent versions of these compilers allow us to flag a region of code with a specific target, so that we can compile only some of the code with support for advanced instructions. Thus in our C++, one might notice macros like `TARGET_HASWELL`. It is then our responsability, at runtime, to only run the regions of code (that we call kernels) matching the properties of the runtime processor. The benefit of this approach is that the compiler not only let us use intrinsic functions, but it can also optimize the rest of the code in the kernel with advanced instructions we enabled.
103+
104+
2. Under Visual Studio, the problem is somewhat simpler. Visual Studio will not only provide the intrinsic functions, but it will also allow us to use them. They will compile just fine. It is at runtime that they may cause a crash. So we do not need to mark regions of code for compilation toward advanced processors (e.g., with `TARGET_HASWELL` macros). The downside of the Visual Studio approach is that the compiler is not allowed to use advanced instructions others than those we specify. In principle, this means that Visual Studio has weaker optimization opportunities.
105+
106+
107+
108+
We also handle the special case where a user is compiling using LLVM clang under Windows, [using the Visual Studio toolchain](https://devblogs.microsoft.com/cppblog/clang-llvm-support-in-visual-studio/). If you compile with LLVM clang under Visual Studio, then the header files (intrin.h or x86intrin.h) no longer provides the intrinsic functions that are unsupported by the processor. This appears to be deliberate on the part of the LLVM engineers. With a few lines of code, we handle this scenario just like LLVM clang under a POSIX system, but forcing the inclusion of the specific headers, and rolling our own intrinsic function as needed.
109+
110+
111+
112+
113+
67114
Regenerating Single Headers From Master
68115
---------------------------------------
69116

@@ -103,7 +150,7 @@ pkg install bash
103150
pkg install cmake
104151
```
105152

106-
You need a recent compiler like clang or gcc. We recommend at least GNU GCC/G++ 7 or LLVM clang 6.
153+
You need a recent compiler like clang or gcc. We recommend at least GNU GCC/G++ 7 or LLVM clang 6.
107154

108155

109156
Building: While in the project repository, do the following:
@@ -153,10 +200,28 @@ We assume you have a common 64-bit Windows PC with at least Visual Studio 2017 a
153200
- Grab the simdjson code from GitHub, e.g., by cloning it using [GitHub Desktop](https://desktop.github.com/).
154201
- Install [CMake](https://cmake.org/download/). When you install it, make sure to ask that `cmake` be made available from the command line. Please choose a recent version of cmake.
155202
- Create a subdirectory within simdjson, such as `VisualStudio`.
156-
- Using a shell, go to this newly created directory. You can start a shell directly from GitHub Desktop (Repository > Open in Command Prompt).
203+
- Using a shell, go to this newly created directory. You can start a shell directly from GitHub Desktop (Repository > Open in Command Prompt).
157204
- Type `cmake -DCMAKE_GENERATOR_PLATFORM=x64 ..` in the shell while in the `VisualStudio` repository. (Alternatively, if you want to build a DLL, you may use the command line `cmake -DCMAKE_GENERATOR_PLATFORM=x64 -DSIMDJSON_BUILD_STATIC=OFF ..`.)
158205
- This last command (`cmake ...`) created a Visual Studio solution file in the newly created directory (e.g., `simdjson.sln`). Open this file in Visual Studio. You should now be able to build the project and run the tests. For example, in the `Solution Explorer` window (available from the `View` menu), right-click `ALL_BUILD` and select `Build`. To test the code, still in the `Solution Explorer` window, select `RUN_TESTS` and select `Build`.
159206

207+
208+
Though having Visual Studio installed is necessary, one can build simdjson using only cmake commands:
209+
210+
- `mkdir build`
211+
- `cd build`
212+
- `cmake ..`
213+
- `cmake --build . -config Release`
214+
215+
216+
Furthermore, if you have installed LLVM clang on Windows, for example as a component of Visual Studio 2019, you can configure and build simdjson using LLVM clang on Windows using cmake:
217+
218+
219+
- `mkdir build`
220+
- `cd build`
221+
- `cmake .. -T ClangCL`
222+
- `cmake --build . -config Release`
223+
224+
160225
### Usage (Using `vcpkg` on 64-bit Windows, Linux and macOS)
161226

162227
[vcpkg](https://github.com/Microsoft/vcpkg) users on Windows, Linux and macOS can download and install `simdjson` with one single command from their favorite shell.

‎doc/basics.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,8 @@ An overview of what you need to know to use simdjson, with examples.
1919
Requirements
2020
------------------
2121

22-
- A recent compiler (LLVM clang6 or better, GNU GCC 7 or better, Visual Studio 2017 or better). We require C++11 support as a minimum.
23-
- A 64-bit system (ARM or x64 Intel/AMD). We run tests on macOS, freeBSD, Linux and Windows. We recommend that Visual Studio users target a 64-bit build (x64) instead of a 32-bit build (x86).
24-
22+
- A recent compiler (LLVM clang6 or better, GNU GCC 7 or better) on a 64-bit (ARM or x64 Intel/AMD) POSIX systems such as macOS, freeBSD or Linux. We require that the compiler supports the C++11 standard or better.
23+
- Visual Studio 2017 or better under 64-bit Windows. Users should target a 64-bit build (x64) instead of a 32-bit build (x86). We support the LLVM clang compiler under Visual Studio (clangcl) as well as as the regular Visual Studio compiler.
2524

2625
Including simdjson
2726
------------------

‎include/simdjson/common_defs.h‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,12 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
4747
#define DEBUG_BLOCK(name, block)
4848
#endif
4949

50-
#if !defined(_MSC_VER) && !defined(SIMDJSON_NO_COMPUTED_GOTO)
50+
#if !defined(SIMDJSON_REGULAR_VISUAL_STUDIO) && !defined(SIMDJSON_NO_COMPUTED_GOTO)
51+
// We assume here that *only* regular visual studio
52+
// does not support computed gotos.
5153
// Implemented using Labels as Values which works in GCC and CLANG (and maybe
5254
// also in Intel's compiler), but won't work in MSVC.
55+
// Compute gotos are good for performance, enable them if you can.
5356
#define SIMDJSON_USE_COMPUTED_GOTO
5457
#endif
5558

@@ -59,7 +62,7 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
5962

6063
#define ISALIGNED_N(ptr, n) (((uintptr_t)(ptr) & ((n)-1)) == 0)
6164

62-
#if defined(_MSC_VER) && !defined(__clang__)
65+
#if defined(SIMDJSON_REGULAR_VISUAL_STUDIO)
6366

6467
#define really_inline __forceinline
6568
#define never_inline __declspec(noinline)
@@ -80,13 +83,7 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
8083
#define SIMDJSON_DISABLE_DEPRECATED_WARNING SIMDJSON_DISABLE_VS_WARNING(4996)
8184
#define SIMDJSON_POP_DISABLE_WARNINGS __pragma(warning( pop ))
8285

83-
#if SIMDJSON_USING_LIBRARY
84-
#define SIMDJSON_DLLIMPORTEXPORT __declspec(dllimport)
85-
#else
86-
#define SIMDJSON_DLLIMPORTEXPORT __declspec(dllexport)
87-
#endif
88-
89-
#else // MSC_VER
86+
#else // SIMDJSON_REGULAR_VISUAL_STUDIO
9087

9188
#define really_inline inline __attribute__((always_inline, unused))
9289
#define never_inline inline __attribute__((noinline, unused))
@@ -119,19 +116,24 @@ constexpr size_t DEFAULT_MAX_DEPTH = 1024;
119116
#define SIMDJSON_DISABLE_DEPRECATED_WARNING SIMDJSON_DISABLE_GCC_WARNING(-Wdeprecated-declarations)
120117
#define SIMDJSON_POP_DISABLE_WARNINGS _Pragma("GCC diagnostic pop")
121118

122-
#if defined(_MSC_VER) && defined(__clang__)
119+
120+
121+
#endif // MSC_VER
122+
123+
#if defined(SIMDJSON_VISUAL_STUDIO)
124+
/**
125+
* It does not matter here whether you are using
126+
* the regular visual studio or clang under visual
127+
* studio.
128+
*/
123129
#if SIMDJSON_USING_LIBRARY
124130
#define SIMDJSON_DLLIMPORTEXPORT __declspec(dllimport)
125131
#else
126132
#define SIMDJSON_DLLIMPORTEXPORT __declspec(dllexport)
127133
#endif
128-
#else
134+
#else
129135
#define SIMDJSON_DLLIMPORTEXPORT
130-
#endif
131-
132-
#endif // MSC_VER
133-
134-
136+
#endif
135137

136138
// C++17 requires string_view.
137139
#if SIMDJSON_CPLUSPLUS17

‎include/simdjson/portability.h‎

Lines changed: 43 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,46 @@
44
#include <cstddef>
55
#include <cstdint>
66
#include <cstdlib>
7+
8+
79
#ifdef _MSC_VER
10+
#define SIMDJSON_VISUAL_STUDIO 1
11+
/**
12+
* We want to differentiate carefully between
13+
* clang under visual studio and regular visual
14+
* studio.
15+
*
16+
* Under clang for Windows, we enable:
17+
* * target pragmas so that part and only part of the
18+
* code gets compiled for advanced instructions.
19+
* * computed gotos.
20+
*
21+
*/
22+
#ifdef __clang__
23+
// clang under visual studio
24+
#define SIMDJSON_CLANG_VISUAL_STUDIO 1
25+
#else
26+
// just regular visual studio (best guess)
27+
#define SIMDJSON_REGULAR_VISUAL_STUDIO 1
28+
#endif // __clang__
29+
#endif // _MSC_VER
30+
31+
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
32+
// https://en.wikipedia.org/wiki/C_alternative_tokens
33+
// This header should have no effect, except maybe
34+
// under Visual Studio.
835
#include <iso646.h>
936
#endif
1037

1138
#if defined(__x86_64__) || defined(_M_AMD64)
12-
#define IS_X86_64 1
39+
#define SIMDJSON_IS_X86_64 1
1340
#endif
1441
#if defined(__aarch64__) || defined(_M_ARM64)
15-
#define IS_ARM64 1
42+
#define SIMDJSON_IS_ARM64 1
1643
#endif
1744

18-
#if (!defined(IS_X86_64)) && (!defined(IS_ARM64))
19-
#if _MSC_VER
45+
#if (!defined(SIMDJSON_IS_X86_64)) && (!defined(SIMDJSON_IS_ARM64))
46+
#ifdef SIMDJSON_REGULAR_VISUAL_STUDIO
2047
#pragma message("The simdjson library is designed\
2148
for 64-bit processors and it seems that you are not \
2249
compiling for a known 64-bit platform. All fast kernels \
@@ -27,7 +54,7 @@ use a 64-bit target such as x64 or 64-bit ARM.")
2754
for 64-bit processors. It seems that you are not \
2855
compiling for a known 64-bit platform."
2956
#endif
30-
#endif // (!defined(IS_X86_64)) && (!defined(IS_ARM64))
57+
#endif // (!defined(SIMDJSON_IS_X86_64)) && (!defined(SIMDJSON_IS_ARM64))
3158
3259
// this is almost standard?
3360
#undef STRINGIFY_IMPLEMENTATION_
@@ -39,26 +66,26 @@ compiling for a known 64-bit platform."
3966
#define SIMDJSON_IMPLEMENTATION_FALLBACK 1
4067
#endif
4168
42-
#if IS_ARM64
69+
#if SIMDJSON_IS_ARM64
4370
#ifndef SIMDJSON_IMPLEMENTATION_ARM64
4471
#define SIMDJSON_IMPLEMENTATION_ARM64 1
4572
#endif
4673
#define SIMDJSON_IMPLEMENTATION_HASWELL 0
4774
#define SIMDJSON_IMPLEMENTATION_WESTMERE 0
48-
#endif // IS_ARM64
75+
#endif // SIMDJSON_IS_ARM64
4976
50-
#if IS_X86_64
77+
#if SIMDJSON_IS_X86_64
5178
#ifndef SIMDJSON_IMPLEMENTATION_HASWELL
5279
#define SIMDJSON_IMPLEMENTATION_HASWELL 1
5380
#endif
5481
#ifndef SIMDJSON_IMPLEMENTATION_WESTMERE
5582
#define SIMDJSON_IMPLEMENTATION_WESTMERE 1
5683
#endif
5784
#define SIMDJSON_IMPLEMENTATION_ARM64 0
58-
#endif // IS_X86_64
85+
#endif // SIMDJSON_IS_X86_64
5986
6087
// we are going to use runtime dispatch
61-
#ifdef IS_X86_64
88+
#ifdef SIMDJSON_IS_X86_64
6289
#ifdef __clang__
6390
// clang does not have GCC push pop
6491
// warning: clang attribute push can't be used within a namespace in clang up
@@ -115,11 +142,10 @@ compiling for a known 64-bit platform."
115142
#define NO_SANITIZE_UNDEFINED
116143
#endif
117144
118-
#ifdef _MSC_VER
119-
#include <intrin.h> // visual studio
120-
#endif
121-
122-
#ifdef _MSC_VER
145+
#ifdef SIMDJSON_VISUAL_STUDIO
146+
// This is one case where we do not distinguish between
147+
// regular visual studio and clang under visual studio.
148+
// clang under Windows has _stricmp (like visual studio) but not strcasecmp (as clang normally has)
123149
#define simdjson_strcasecmp _stricmp
124150
#else
125151
#define simdjson_strcasecmp strcasecmp
@@ -129,7 +155,7 @@ namespace simdjson {
129155
/** @private portable version of posix_memalign */
130156
static inline void *aligned_malloc(size_t alignment, size_t size) {
131157
void *p;
132-
#ifdef _MSC_VER
158+
#ifdef SIMDJSON_VISUAL_STUDIO
133159
p = _aligned_malloc(size, alignment);
134160
#elif defined(__MINGW32__) || defined(__MINGW64__)
135161
p = __mingw_aligned_malloc(size, alignment);
@@ -153,7 +179,7 @@ static inline void aligned_free(void *mem_block) {
153179
if (mem_block == nullptr) {
154180
return;
155181
}
156-
#ifdef _MSC_VER
182+
#ifdef SIMDJSON_VISUAL_STUDIO
157183
_aligned_free(mem_block);
158184
#elif defined(__MINGW32__) || defined(__MINGW64__)
159185
__mingw_aligned_free(mem_block);

‎singleheader/amalgamate.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22
########################################################################
33
# Generates an "amalgamation build" for roaring. Inspired by similar
44
# script used by whefs.

‎singleheader/amalgamate_demo.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
/* auto-generated on Thu Apr 23 15:36:58 PDT 2020. Do not edit! */
1+
/* auto-generated on Mon 27 Apr 2020 21:20:37 EDT. Do not edit! */
22

33
#include <iostream>
44
#include "simdjson.h"

0 commit comments

Comments
 (0)