OK, let me put down how I see it. It might make a difference, it might not.
Whether we like it or not, Wedge isn't a forum. It's a framework first and foremost, that just happens to be tuned to running a forum. User authentication, the template system, plugin system, these are all symptoms of a framework. As are a good deal of the components actually in Wedge.
Now, from my perspective, if it is the case that we have this setup, then to me the core framework should be best optimised for the one thing that is common to all forums (and varies between important and unimportant for non-forum uses): making the process of being able to share content and discussion easier.
Having AeMe in the core does this, IMO, both the auto embed and gallery parts (assuming using them for attachments), but the calendar and birthdays facilities do not inherently make it easier to share content and discussion.
To me, they're facilitators for social engagement, not content and discussion, so while they are important for some forums, for others they're absolutely unimportant.
Now, you could argue that other things we're doing fall into the same category, in particular the ability to like content, but that to me sits on the border between being content encouraging and social participation. It allows people to indicate they like some content, encourages more of it as a consequence and it allows people to feel they have an outlet for expressing an opinion without necessarily having to post - on those grounds I'd be inclined to make it core rather than through a plugin.
The bottom line is that this is a complex road we travel, we're not going to be able to please everyone all of the time, but I suspect the facility of having things as plugins that can co-exist nicely will really help matters.