It seems in the more recent Synology DSM (6+) that the transmission package installs with a user account (i.e UID >1024), and every reboot the system sets the transmission user in/etc/passwd to have the shell /sbin/nologin - its supposed to be /bin/sh. This causes the service to be unable to start, due to a lack of shell.
I fixed it by moving the UID of the transmission user to < 1024 - it was 1028 so I moved it to 128:
- update /etc/passwd UID number in the transmission users' line
- chown relevant files (chown looks up UID using /etc/passwd so gets the updated number):
- chown -R transmission /usr/syno/etc/preference/transmission
- chown -R transmission /volume1/\@appstore/transmission
reboot and the service started on boot #win.