how to debug avx512 gfni erasure nasm code in Linux?

[jeffareid]

Currently I'm using Visual Studio 2026 with masm or nasm under Windows 11 Pro. If needed, I can debug in disassembly mode, one instruction at a time with the registers displayed in one of the debugger windows (I usually choose the zmm registers for this).

[pablodelara]

Hi @jeffareid. You can do it using gdb as long as the system supports AVX512 and GFNI ISA. Any issues that you are encountering?

[jeffareid]

GDB doesn't work with VS Code for debugging. This issue has been around for a few years:

GDB - fail to open terminal

It's gotten worse, usually locking up, and preventing a program from being debugged. In some cases, it would take a few minutes for the target program to start, in other cases, the target program never starts.

CodeLLDB works, but there is no register window. Instead you have to go to debug console and enter commands to display the registers. One of the IDE | debugger tools might work, but I don't know which ones would support AVX512 | GFNI, GCC and NASM. This is why I created this issue, as I'm wondering what toolset, if any, that others here might be using that would provide a source level interface than include a window to display register values like Visual Studio.

[pablodelara]

Oh ok, so you meant "Windows" on your title.

[jeffareid]

Oh ok, so you meant "Windows" on your title.

Windows | Visual Studio works fine and is what I use now. The GDB issue is related to VS Code, and I'm not sure why someone would use VS Code on Windows. There's no Visual Studio for Linux, and I'm wondering if there is something similar for Linux.

[jeffareid]

Microsoft has some advanced development tools. Using the SDK (software development kit) and DDK (device driver kit aka device development kit) involves two systems, a host system that runs the IDE, and the target system that is being debugged. It is similar to Visual Studio, but has the advantage that it runs at ring 0 kernel level on the target system, which is why it used for device drivers, allowing interrupt routines to be debugged. The DDK was renamed to WDK (Windows driver kit):

Windows Driver Kit

The only time I used it was to work on the ATAPI driver for Windows NT 4.0 (so a long time ago), as I worked for a tape company, and some of our tape drives used the ATAPI interface, and it was a joint effort to get our tape drives running on Windows NT.

[jeffareid]

To clarify, I posted that link to show it has been an issue with VS Code on both Linux and Windows. Do a web search for

ubuntu vscode gdb failed to set controlling terminal

and you'll get a lot of hits. If you keep doing this using google search eventually the AI will suggest switching to CodeLLDB. The last part of this reads:

Key Configuration Fields:
type: Must be set to "lldb" to use the CodeLLDB extension
program: Ensure this points exactly to your compiled executable
file.terminal: Setting this to "integrated" forces the program's input/output to stay inside the built-in VS Code terminal panel.

Once these steps are completed, press F5 to start debugging smoothly without the GDB terminal errors.

---

This will get C | C++ debugging to work on Linuz, but additional steps are needed for debugging assembly code on Linux.

[jeffareid]

Getting back to the original question, what tool sets do programmers use to debug AVX512 + GFNI on Linux (or do they not use Linux)?

[pablodelara]

I generally debug on Linux, using gdb. But I use either Makefile or CMake to build, not VSCode.

[jeffareid]

I generally debug on Linux, using gdb. But I use either Makefile or CMake to build, not VSCode.

How difficult is it to use gdb to display zmm registers while stepping through assembly or disassembled (in case of macros) code?