I know why it's complex, but if you turn it round a bit, it makes sense.
The core attachment system has two principle weaknesses, in my book: firstly, it's not extensible, and secondly its management capabilities are limited.
Really, all an attachment is is a file uploaded to the system... Post attachments are restricted by the topic permissions, album attachments by the album permissions. The more you look, the clearer it is that they have an awful lot in common, probably more than is realised.
And I'm not really seeing the problems with doing it, other than the fear of doing it. All the issues related to how I think it can be made extensible aren't really issues at all, I already covered off how that would likely work, at least in my head, by giving each item a source to have originated from (both a place and a contextual id, such as album+1 for an item in album 1 or help desk+2 for an item attached to a helpdesk ticket, etc.)