...the OP’s own answer is wrong in so far, that there are no “special system permissions”. – The “no permission” problem boils down to ... no permissions.
Unfortunately it is not easy to debug, because adb makes it a secret which device it tries to access! On Linux, it tries to open the “USB serial converter” device of the phone, which is e.g. /dev/bus/usb/001/115 (your bus number and device address will vary). This is sometimes linked and used from /dev/android_adb.
lsusb will help to find bus number and device address. Beware that the device address will change for sure if you re-plug, as might the bus number if the port gets confused about which speed to use (e.g. one physical port ends up on one logical bus or another).
An lsusb-line looks similar to this: Bus 001 Device 115: ID 4321:fedc bla bla bla
lsusb -v might help you to find the device if the “bla bla bla” is not hint enough (sometimes it does neither contain the manufacturer, nor the model of the phone).
Once you know the device, check with your own eyes that ls -a /dev/bus/usb/001/115 is really accessible for the user in question! Then check that it works with chmod and fix your udev setup.
PS1: /dev/android_adb can only point to one device, so make sure it does what you want.
PS2: Unrelated to this question, but less well known: adb has a fixed list of vendor ids it goes through. This list can be extended from ~/.android/adb_usb.ini, which should contain 0x4321 (if we follow my example lsusb line from above). – Not needed here, as you don’t even get a “no permissions” if the vendor id is not known.