UHK60: Windows 11 doesn't receive input occasionally

[treedy]

I've have an issue where my UHK 60 is on and appears to respond to key presses, but Windows doesn't receive or acknowledge the input. Usually, I unplug and plug back in the USB cable and I don't have any issues for a couple of hours to maybe a few weeks. When it exhibited this behavior today, I ran UHK Agent. After a few seconds it displayed "[DeviceOperation] ValidatedUserConfig from eeprom error". Agent crashed a couple of times, but on the 3rd run it didn't crash. I hit my reboot macro as suggested in UltimateHackingKeyboard/agent#2911. I briefly saw a message stating something along the lines of "it looks like your keyboard has an error, we can fix it".

I have attached the agent log. If there's more information I can provide, I'll do my best to get it to you. Unfortunately, this isn't an easily reproducible bug.

FW Version: 17.0.0
Agent Version: 10.0.0
Windows 11

[kareltucek]

Ah. Sorry for late reply!

Indeed this seems complicated to troubleshoot.

Anyways, to get to logs, you will need the following. (Or if you wait for a bit, they will be soon (weeks) released as normal releases.):

• nightly build of Agent; Any master CI build should work, e.g.: https://github.com/UltimateHackingKeyboard/agent/actions/runs/27467679109
  • contains new zephyr shell components
• staging build of firmware, e.g.: https://github.com/UltimateHackingKeyboard/firmware/actions/runs/27680390823
  • contains some logging support (support for the new logging Agent machinery, and will log usb failures including reason codes)
  • is build on an updated sdk, which may resolve your issue by itself.
• check how to access logs at https://uhk.io/knowledgebase/how-to-report-firmware-issues

Uhk60 has very little logging, so don't worry if you don't see much.

When you manage to reproduce it, go to the log section, tap a few keys and see if anything appears.

[kareltucek]

I have now published the above Agent with a 17.2.0 firmware at https://github.com/UltimateHackingKeyboard/agent/releases/tag/v10.1.0

[treedy]

I just had the issue happen again today. When Windows wasn't responding to keyboard presses, I launched Agent. It crashed. I relaunched the agent and I received the [DeviceOperation] ValidatedUserConfig from eeprom error message (like the previous times). It crashed again. I relaunched and received an Can't write to device message (I forgot to copy the actual message, but it was short and similar meaning). Clicking on the Agent logo didn't bring up the "advanced settings" option like it does when the keyboard is working.

Using the new Agent and 17.2.0 firmware

[kareltucek]

Right. Pulling logs over usb when usb doesn't work, is impractical.

Can you verify whether the windows still sees UHK as connected and functional? (I guess it does not.)

https://claude.ai/share/e1ed26ba-7d11-4f55-be36-0f7eb32d85df

[kareltucek]

Another development. For now, I assume that the usb interface goes down entirelly, meaning that the above logging approach won't work, as it works over usb. I have put together a firmware that dumps some info into the status buffer, then reboots and allows retrieving the info after the reboot with a working usb.

Please:

• flash the firmware from https://github.com/UltimateHackingKeyboard/firmware/actions/runs/28674090096
• map a diagnose usb macro somewhere convenient.
• try it so you know what to expect - a yellow bar should pop up

Once the issue occurs again, tap the macro, gather the log, post it here.

[treedy]

I flashed the firmware, mapped the macro, and gave it a test run. I saw the yellow diagnostic messages. I'll update when this happens again.

[treedy]

@kareltucek So the issue happened this morning. I opened the UHK Agent and the LED started displaying "OUT". The yellow debug bar showed

Usb_RemoteWakeup
HID report dropped: EP 83 transfer cancelled
HID report dropped: EP 83 transfer cancelled
...

I then pressed my macro for diagnose usb. I heard the Windows 11 sound for USB connection. Agent then displayed the following in the yellow debug bar

transfer cancelled
HID report dropped: EP 83 transfer cancelled
HID report dropped: EP 83 transfer cancelled
c2usb diag (t=333747172 ms):
  power=L0-on configured=1 active=1
  last ok transfer: EP 83, 546 ms ago (42504 total)
  EP error: total=31 sinceOk=1, last: EP 83 err=0 506 ms ago
  ctrl stall: total=181 sinceOk=1, last: EP 00 err=0 377 ms ago
  EP busy reject: total=1 sinceOk=0, last: EP 83 err=0 109145 ms ago
  bus event: suspend x1, last 333746918 ms ago
  bus event: resume x1, last 333746708 ms ago
  bus event: suspend x1, last 333746705 ms ago
  bus event: resume x1, last 333745206 ms ago
  bus event: reset x1, last 333745206 ms ago
  bus event: suspend x1, last 324715933 ms ago
  bus event: resume x1, last 260910301 ms ago
  bus event: suspend x1, last 238165332 ms ago
  bus event: resume x1, last 178092969 ms ago
  bus event: suspend x1, last 166147654 ms ago
  bus event: resume x1, last 98010572 ms ago
  bus event: suspend x1, last 66088945 ms ago
  bus event: resume x1, last 2875588 ms ago
  bus event: suspend x1, last 214487 ms ago
  bus event: resume x1, last 214460 ms ago
  bus event: reset x352, last 436 ms ago
Printing trace buffer because: Diagnostics reboot.
ID: 2, EV: 67109509, Tag: c1873a8, Time: 333747174
=== K22P121M120SF7 Reset Cause Debug ===
RCM_SRS0:

The Windows 11 still wasn't receiving input from the UHK60. However, the Agent icon on the LED was on and "ERR" displayed.

I then pressed my keyboard reboot macro and Agent then displayed the following in the yellow debug bar

0x82
RCM_SRS1:  0x00
RCM_SSRS0: 0x00
RCM_SSRS1: 0x00
PMC_LVDSC1: 0x10
PMC_LVDSC2: 0x00
PMC_REGSC:  0x04
SCB_HFSR:  0x00000000
SCB_CFSR:  0x00000000
SCB_BFAR:  0xE000ED38
SCB_MMFAR: 0xE000ED34

=== Reset Cause Analysis ===
Power-On Reset detected
Loss of Clock Reset detected
LVDSC1 0x10     Low Voltage Reset Enabled
RSGSC 0x04      Regulator is in run regulation
Trace:
}{}{}{}{}{}<i4>{}{}{}{}{}

The "ERR" went away and Windows is receiving input from the keyboard.

Separate, minor feedback on the yellow debug bar: a mouse action (a button or right-click) to copy its content would be great.

[kareltucek]

Nice!

So at least the command interface remained at least partially up. Sufficiently enough for Agent to communicate with the UHK.

The log says that the host has been resetting the bus for the last 3 minutes twice a second.

Can you dig more into what is going on from Windows's point of view please?

@benedekkupper do you have any insights?

[treedy]

In Windows' Microsoft-Windows-Kernel-PnP/Device Management event log, I see the following around the times the keyboard has stopped working:

Device USB\VID_37A8&PID_0003\1029247501 has been surprise removed as it is reported as missing on the bus.
Count of devices removed: 9

FWIW: There's nothing in the past few days in setupapi.dev.log nor in the Event Log's "Hardware Events".

I'll add some scripts that I can double click on my desktop the next time it goes down to gather the "live" status of USB connections as well as the USBView tool. I have now enabled Microsoft-Windows-USB-USBPORT and Microsoft-Windows-USB-USBHUB3 Event Logs.

[benedekkupper]

The only thing I can see from the logs are transfer problems on the Agent channel, not the keyboard functionality itself. Those errors are rather strange themselves, as these "transfer cancelled" events are triggered only on USB_DeviceKhciCancel calls, which are only called when the endpoint is closed, i.e. at USB reset. I have reworked the stack so these cancellations are propagated, in case this happens on the key sending endpoints too: IntergatedCircuits/c2usb@7654d47 @kareltucek maybe you can check this out.

For me some WireShark logs from @treedy would be best, but also it would be great if the UHK's usb diag contained the SET_CONFIGURATION event as well (@kareltucek). Another thing that would help us a lot if you are aware if there is an earlier known good FW version that doesn't produce this behavior for you @treedy?

[kareltucek]

Here is updated firmware: https://github.com/UltimateHackingKeyboard/firmware/actions/runs/29152747791

• Rebased onto Benedek's rework
• added the CONFIGURATION and DECONFIGURATION event traces.

Edit: the linked firmware bricks the keyboard. I will post a fixed one.

[kareltucek]

@benedekkupper the build I have posted bricks the uhk pretty much.

I have retested today with this branch: https://github.com/UltimateHackingKeyboard/firmware/tree/usb_reality_check

Changes on the branch w.r.t. uhk master:

• uses the linked c2usb commit
• firmware/patches/etl/0001-delegate-allow-constructing-from-non-capturing-lambd.patch with conflicts resolved by claude.

Excerpt from dmesg:

[27915.659485] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[27915.866472] usb 1-2.4.1: Device not responding to setup address.
[27916.075348] usb 1-2.4.1: device not accepting address 124, error -71
[27916.075573] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27916.174366] usb 1-2.4.1: new full-speed USB device number 125 using xhci_hcd
[27916.271477] usb 1-2.4.1: device descriptor read/64, error -32
[27916.468733] usb 1-2.4.1: device descriptor read/64, error -32
[27916.570849] usb 1-2.4-port1: attempt power cycle
[27917.198395] usb 1-2.4.1: new full-speed USB device number 126 using xhci_hcd
[27917.198568] usb 1-2.4.1: Device not responding to setup address.
[27917.403521] usb 1-2.4.1: Device not responding to setup address.
[27917.611408] usb 1-2.4.1: device not accepting address 126, error -71
[27917.611658] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27917.699419] usb 1-2.4.1: new full-speed USB device number 127 using xhci_hcd
[27917.699576] usb 1-2.4.1: Device not responding to setup address.
[27917.906820] usb 1-2.4.1: Device not responding to setup address.
[27918.114562] usb 1-2.4.1: device not accepting address 127, error -71
[27918.114800] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27918.114870] usb 1-2.4-port1: unable to enumerate USB device
[27958.402820] usb 1-2.4.1: new full-speed USB device number 7 using xhci_hcd
[27958.502729] usb 1-2.4.1: New USB device found, idVendor=1d50, idProduct=6123, bcdDevice= 0.02
[27958.502754] usb 1-2.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[27958.502761] usb 1-2.4.1: Product: UHK 60 v2 Bootloader
[27958.502767] usb 1-2.4.1: Manufacturer: Ultimate Gadget Laboratories
[27958.513534] hid-generic 0003:1D50:6123.00E4: hiddev0,hidraw1: USB HID v1.00 Device [Ultimate Gadget Laboratories UHK 60 v2 Bootloader] on usb-0000:00:14.0-2.4.1/input0
[27963.014066] usb 1-2.4.1: USB disconnect, device number 7
[27963.219776] usb 1-2.4.1: new full-speed USB device number 9 using xhci_hcd
[27967.920678] usb 1-2.4.1: Device not responding to setup address.
[27968.125012] usb 1-2.4.1: Device not responding to setup address.
[27968.332073] usb 1-2.4.1: device not accepting address 9, error -71
[27968.332480] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27968.648981] usb 1-2.4.1: new full-speed USB device number 11 using xhci_hcd
[27968.761776] usb 1-2.4.1: New USB device found, idVendor=1d50, idProduct=6123, bcdDevice= 0.02
[27968.761799] usb 1-2.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[27968.761808] usb 1-2.4.1: Product: UHK 60 v2 Bootloader
[27968.761814] usb 1-2.4.1: Manufacturer: Ultimate Gadget Laboratories
[27968.769516] hid-generic 0003:1D50:6123.00E5: hiddev0,hidraw1: USB HID v1.00 Device [Ultimate Gadget Laboratories UHK 60 v2 Bootloader] on usb-0000:00:14.0-2.4.1/input0
[27973.253684] usb 1-2.4.1: USB disconnect, device number 11
[27973.461070] usb 1-2.4.1: new full-speed USB device number 12 using xhci_hcd
[27978.637271] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
[27978.845347] usb 1-2.4.1: Device not responding to setup address.
[27979.052246] usb 1-2.4.1: device not accepting address 12, error -71
[27979.052476] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27979.140298] usb 1-2.4.1: new full-speed USB device number 13 using xhci_hcd
[27979.229301] usb 1-2.4.1: device descriptor read/64, error -32
[27979.422283] usb 1-2.4.1: device descriptor read/64, error -32
[27979.525567] usb 1-2.4-port1: attempt power cycle
[27980.151291] usb 1-2.4.1: new full-speed USB device number 14 using xhci_hcd
[27980.151428] usb 1-2.4.1: Device not responding to setup address.
[27980.356612] usb 1-2.4.1: Device not responding to setup address.
[27980.565287] usb 1-2.4.1: device not accepting address 14, error -71
[27980.565495] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27980.653282] usb 1-2.4.1: new full-speed USB device number 15 using xhci_hcd
[27980.653399] usb 1-2.4.1: Device not responding to setup address.
[27980.861423] usb 1-2.4.1: Device not responding to setup address.
[27981.068316] usb 1-2.4.1: device not accepting address 15, error -71
[27981.068587] usb 1-2.4.1: WARN: invalid context state for evaluate context command.
[27981.068695] usb 1-2.4-port1: unable to enumerate USB device

Does that tell you anything? Should I dig deeper?