Here’s what I did for sway, though I agree it took me a while to arrive at this solution:

In the file .config/sway/config:

input "type:keyboard" {
  # Use the following to remap single keys.
  # Weird letter codes are in /usr/share/X11/xkb/keycodes/evdev
  # Keysym names are in /usr/include/X11/keysymdef.h
  # xkb moded layout is in .xkb/symbols/qwerty_prtsc_to_menu in this repo
  #
  xkb_layout "qwerty_prtsc_to_menu"
  xkb_options caps:super
  repeat_delay 220
  repeat_rate 70
}

in the file .xkb/symbols/qwerty_prtsc_to_menu:

default partial alphanumeric_keys
xkb_symbols "basic" {
  include "us"
  replace key "replacewithlessthansymbol"PRSC"replacewithgreaterthansymbol" { [ Menu ] };
};

I’m pretty sure you can do xkb_layout with input device matching in hyprland though I stopped using hyprland a while back.

Some things like caps:super already exist in xkb_options, but less common ones require manually creating a layout. I found an xkb doc describing all these when I was working on this but I can’t find it now. If I manage to locate it I’ll update this comment.

EDIT: For the qwerty_prtsc_to_menu file above it’s suppose to be the actual less than and greater than symbols however lemmy’s code formatting is removing them when I save the comment, sorry about that

EDIT2: I didn’t realize that you wanted a momentary layer, so this solution will not work for you exactly. However I believe this is possible with keymap switching like described here https://wiki.archlinux.org/title/sway#Keymap in addition to what I have suggested above.

Sooooo … There is an update on this rant – or a solution. I took the time and learned the absolute minimum and basics of xkb. And after a lot of crashes with proper error messages and trying quite some time I was able to replicate my current .Xmodmap using xkb and implement it as custom layout.

Since sharing is caring, here is the fully documented file.

// vim:syntax=xkb


// Manually cobbled together using those sources:
//
// https://github.com/hardfau1t/dotfiles/blob/main/xkb/.config/xkb/symbols/cust
// https://github.com/swaywm/sway/issues/4250
// https://www.charvolant.org/doug/xkb/html/node5.html
// The file /usr/share/X11/xkb/symbols/us
//
// Set up for and tested with a Dell XPS 13 9380 laptop, but it should work
// With any keyboard variant. Just include the correct variant and change the
// keys accordingly.
//
// Place this in the file `${HOME}/.xkb/symbols/custom-bindings`
//
// For Hyprland set this in the config (other compositors or WMs might have
// something similar where you can define the keyboard layout):
//
//     input {
//        kb_layout = custom-bindings
//        [more/other input configuration]
//     }
//
// NOTE: If it crashes, some of the configuration is wrong or could not be
//       properly loaded. Unfortunately there aren’t useful logs. Try to
//       comment out individual parts of this file until it stops crashing,
//       then start over from there adding one section/line after another
//       and reload and test the configuration.
//
// GOOD LUCK! :)


// Layout definition
//
// The layout uses the basic US layout and the level3 ralt_switch to make the
// right ALT key actually usable as level3 switch (referred to as AltGr from
// now on).
//
// All the other includes are referring to this file. If you do not use the US
// layout, make sure to adapt the key definitions in the other sections.
//
// In `/usr/share/X11/xkb/symbols` you’ll find the variants. This file uses the
// variant `us`. Again: Just open your matching symbols file, there you’ll see
// all the sections in similar syntax, just take from there what you need.
default partial alphanumeric_keys
xkb_symbols "basic" {
    name[Group1]= "English (US, RALT, customizations)";
    include "us"
    include "level3(ralt_switch)"
    include "custom-bindings(umlauts)"
    include "custom-bindings(quotation-marks)"
    include "custom-bindings(punctuation)"
    include "custom-bindings(script-numbers)"
};


// German umlauts
//
// The corresponding base letters with AltGr and Caps+AltGr return the umlaut
// types of that letter. Also the sparsely used “uppercase ß” is available
// with AltGr+Caps+s
//
// äÄ   öÖ   üÜ   ßẞ
xkb_symbols "umlauts" {
    key <AC01> {[  a , A , adiaeresis , Adiaeresis  ]};
    key <AD09> {[  o , O , odiaeresis , Odiaeresis  ]};
    key <AD07> {[  u , U , udiaeresis , Udiaeresis  ]};
    key <AC02> {[  s , S , ssharp     , U1E9E       ]};
};


// Quotation marks
//
// z and x pressed with AltGr or AltGr+Caps create outer („“) and inner (‚‘)
// German quotation marks.
//
// c and v do the same for the English quotation marks (“” and ‘’).
//
// z, x = „“ & ‚‘   c, v = “” & ‘’
xkb_symbols "quotation-marks" {
    key <AB01> {[  z , Z , doublelowquotemark   , singlelowquotemark    ]};
    key <AB02> {[  x , X , leftdoublequotemark  , leftsinglequotemark   ]};
    key <AB03> {[  c , C , leftdoublequotemark  , leftsinglequotemark   ]};
    key <AB04> {[  v , V , rightdoublequotemark , rightsinglequotemark  ]};
};


// Punctuation characters
//
// When pressed with AltGr the following symbols are returned.
//
// -   “Halbgeviertstrich”, a type of dash used as replacement for various
//     other punctuation in German (–), identical to an “en dash” character.
//
// +   typographically correct apostrophe (’)
//
// .   Ellipsis character (…).
xkb_symbols "punctuation" {
    key <AE11> {[  minus      , underscore , U2013     ]};
    key <AC11> {[  apostrophe , quotedbl   , U2019     ]};
    key <AB09> {[  period     , greater    , ellipsis  ]};
};


// Subscript and superscript numbers
//
// When pressing the numbers from the numbers row with AltGr or AltGr+Caps,
// numbers are returend in subscript (AltGr) or superscript (AltGr+Caps).
//
// AltGr+Number      = Text ₁₂₃₄₅₆₇₈₉₀ more text
// 
// AltGr+Caps+Number = Text ¹²³⁴⁵⁶⁷⁸⁹⁰ more text
xkb_symbols "script-numbers" {
    key <AE01> {[  1, exclam      , U2081 , U00B9  ]};
    key <AE02> {[  2, at          , U2082 , U00B2  ]};
    key <AE03> {[  3, numbersign  , U2083 , U00B3  ]};
    key <AE04> {[  4, dollar      , U2084 , U2074  ]};
    key <AE05> {[  5, percent     , U2085 , U2075  ]};
    key <AE06> {[  6, asciicircum , U2086 , U2076  ]};
    key <AE07> {[  7, ampersand   , U2087 , U2077  ]};
    key <AE08> {[  8, asterisk    , U2088 , U2078  ]};
    key <AE09> {[  9, parenleft   , U2089 , U2079  ]};
    key <AE10> {[  0, parenright  , U2080 , U2070  ]};
};