Soft merging of posts

Nao

  • Dadman with a boy
  • Posts: 16,079
Soft merging of posts
« on July 24th, 2013, 11:07 AM »
Better temperatures here today, so I figured I could get more work done on Wedge...

In terms of innovations, I'm not particularly fond of trying to figure out new ones, but I have to say it feels good to do things that I haven't seen done elsewhere, and I'm trying not to get stopped by 'reality', and do things my own way.

So, one of the things that have always bothered me, is merging of double-posts. It's a feature I've loved since day one (as you may remember from my stint on the 'Merge double posts' mod on sm.org, and generally adding the same feature in here by default), but that was in contradiction with another feature I wanted in: threaded view.

The trouble with threadles

Threaded view, as controversial it is, as 'outdated' it might be to some (and I'm not saying it isn't), implies that if you answer every single post in a thread, you'll end up with a lot of double posts, but if you start merging them, you'll end up with a long post that loses its association to the thread parents, and that is, if you'll forgive me, absolutely FUCKED UP.

I started considering doing a 'soft' merging of these posts, i.e.: the posts are not physically merged in the database, and instead they're only merged in the template if there are two or more posts by a same user in the same level.

Skeletons in the closet

Here's where it gets complicated. Ever since I introduced mini-skeletons for topic posts, it became harder to do anything like rendering only parts of a post. I made a modification to the skeleton class, where you could specifically ask it to render just a single layer from a skeleton, and I still have that code around (uncommitted), but it was getting too complex, even for me, and I wrote the whole damn thing... Essentially, it's too fragile.

I'm still considering doing it this way, though, because it's so 'clean'. But the code changes, alas, wouldn't be... So, this morning, I figured I'd try doing it with my favorite little hobby...

Ne vous déplaise, en dansant la Javanaise... ♫

And it worked. I wrote 10 lines of JavaScript, about 370 unminified bytes, and it does exactly what I wanted.
Actually, I even tested it on wedge.org, so you can get a feel of it...

http://wedge.org/pub/feats/6108/new-revs/2160/

As you can see, all of the double posts are now merged into one, and they all retain their respective action bars. Really, it's just all about removing the duplicate signature and user box areas.

It takes one to know one.

There are a few caveats to this technique, and I'm weary of committing my code, so here's the list of problems, and possible fixes, and I'd really, REALLY appreciate if you guys could share your feelings about this implementation, and whether it's worth it.

- Since it executes at render time, you can for a very short time see the unmodified posts. I've fixed this by putting the code into the script.js file, instead of topic.js, as it executes earlier, but it's not exactly a noticeable improvement, so I'll probably end up putting it back into topic.js later, and anyway, if you don't actually refresh the page, it's unlikely you'll notice the difference when simply browsing through pages, which is what you'll be doing most of the time, anyway.

- This doesn't fix postbg/postbg2 alternance. Well, I'll fix that one next... But, more importantly, because it's JS, it means that if you do alternances differently (or not at all) in your version, it'll be broken by my upcoming fix.

- Anchor links are broken. e.g., if you're in a soft-merged post, and you click the subject, you'll end up on the same page, but at the top, because it won't find the message ID, which was suppressed by JS. A possible fix would be to just move the message ID to the postheader for the corresponding soft-merged post, but it'll also break any code that accesses .root and then gets the message ID through this. Very, very bad idea either way... I need to fix this one, ASAP.

- I haven't tested all action menu options, maybe some will be broken, potentially by the change just above this one.

So... Thoughts, please..? Or just Likes :P

Aaron

  • Posts: 356
Re: Soft merging of posts
« Reply #1, on July 24th, 2013, 12:27 PM »
Love the idea! I have to wonder, though, why tackle this with JavaScript at all? It seems like something that can easily be handled in the topic template instead — which also makes solving the issues you list much easier.
"The entire British Empire was built on cups of tea … and if you think I'm going to war without one, mate, you're mistaken."

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Soft merging of posts
« Reply #2, on July 24th, 2013, 12:38 PM »
Quote from Aaron on July 24th, 2013, 12:27 PM
Love the idea! I have to wonder, though, why tackle this with JavaScript at all? It seems like something that can easily be handled in the topic template instead — which also makes solving the issues you list much easier.
As I explained in 'Skeleton in the closet', Wedge's mini-skeleton system, while VERY flexible, is built for show its skeletons one by one... Obviously. As in Wedge, a message has its own skeleton[1], it would force me to 'disrupt' the skeleton rendering flow in the middle, which is both going to make the process slower, and in total contradiction with the philosophy of the very system I built... :-/ (which is, basically: give as much flexibility to third-party devs, but within the bounds of a normal flow.)

Oh, which makes me think, I forgot to mention one of the issues:

- If you build a theme that doesn't use the same structure for the message mini-skeleton, at all, then you're screwed... Maybe I'd have to add an optional global var, declared in index.template.php, saying "dont_soft_merge", or something... Then it'd be okay. Except if a plugin starts moving layers around, of course... But then it never ends, eh..? And it can always add the variable manually, too.

Regarding doing it all in PHP, I think that it's still doable, but perhaps instead at ob_sessrewrite time, and it wouldn't be very easy to manipulate the DOM at this point, so... I'm not thrilled with the idea.
Any ideas are welcome, of course.
 1. If you're not aware of what a mini-skeleton is, I'll advise you to REALLY, really read about them. There's a topic about them. I'll try to document them properly in the future, but it's something I spent weeks on, so I kinda assume that anyone interested in Wedge, had already read it by that point... ;) But I'm not sure whether you'd rather follow Wedge or ElkArte these days, so I'm going to assume you left the project months ago and only came back today to check it out, or something.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Soft merging of posts
« Reply #3, on July 24th, 2013, 05:47 PM »
When I first saw it in one of the other topics, I was like... is that a bug? It's neat, and yes, threaded does pretty much demand not physically merging. (Though I still don't see how we fix the various UX issues with threaded even if the technical implementation isn't a killer. Show me a site that actually does it well... Reddit doesn't count because it looks and acts like a mess, sorry.)

I like what you've done, once I realised it was supposed to be like that ;) Doing it at JS time is an interesting solution and does generally make it cleaner on all levels (knowing the skeleton stuff etc.) though I can argue there is a usability issue if you have a run of posts from one person, e.g. like in New Revs, and the user info is way way way back up... though that's largely true for all long posts anyway.

And yes, tampering with it at the skeleton level would be incredibly messy, though I daresay it would be possible. I would expect it to become very convoluted if so done, and it's already got a little bit of a barrier to entry for modders (though once they get round the idea of skeletons and miniskeletons, which they'll have to for pretty much any plugin, it's not so much a big dea), so making it more complex is not a good idea.

I do wonder what will happen if themes (which modify templates, rather than skins which just modify CSS) come along and change that up. Perhaps the variable is a good idea so that a theme can indicate it as such.
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Soft merging of posts
« Reply #4, on July 24th, 2013, 06:50 PM »
Quote from Arantor on July 24th, 2013, 05:47 PM
When I first saw it in one of the other topics, I was like... is that a bug? It's neat, and yes, threaded does pretty much demand not physically merging. (Though I still don't see how we fix the various UX issues with threaded even if the technical implementation isn't a killer. Show me a site that actually does it well... Reddit doesn't count because it looks and acts like a mess, sorry.)
It's not that I absolutely WANT to make a threaded view absolutely... It's that I believe it might be helpful to some people, and it's not that hard to implement. I'm more curious about whether I should add a 'limit' to the threading level, e.g. stackoverflow has a limit of one level of replies after a comment, which makes a lot of sense, if you'll allow me. (The guy even documented this conscious choice on his blog, and I agree with it... Though I think it shouldn't be hardcoded, really.)
Quote
I like what you've done, once I realised it was supposed to be like that ;) Doing it at JS time is an interesting solution and does generally make it cleaner on all levels (knowing the skeleton stuff etc.) though I can argue there is a usability issue if you have a run of posts from one person, e.g. like in New Revs, and the user info is way way way back up... though that's largely true for all long posts anyway.
Yeah... There are always going to be a few usability issues, but this one could be solved in a bold way, by making the user info box fixed when you're scrolling... A bit like the sidebar contents, but in real time. Of course, this creates an issue in that we then need to have a .scroll() event, and calculate our page position, and the position of all user boxes on each scroll, which is, err... Probably a bit costly, I guess..?
Quote
And yes, tampering with it at the skeleton level would be incredibly messy, though I daresay it would be possible.
I said it-- it's possible. It's just plain ugly.
I've been trying to find a non-ugly solution, but it always involves manipulating the DOM after the render phase, and... It's something I'm not too keen trying to do, eh.
Quote
I would expect it to become very convoluted if so done, and it's already got a little bit of a barrier to entry for modders (though once they get round the idea of skeletons and miniskeletons, which they'll have to for pretty much any plugin, it's not so much a big dea),
I'm expecting 99% of all mini-skeleton uses will be in the main codebase, anyway... Plugins and themes, lemme guess... Will simply expand upon the existing ones, eh.
Quote
so making it more complex is not a good idea.
So, no need to commit my 'render only this layer' portion of the skeleton code, I suppose..?
Quote
I do wonder what will happen if themes (which modify templates, rather than skins which just modify CSS) come along and change that up. Perhaps the variable is a good idea so that a theme can indicate it as such.
That's the idea.

Oh, I implemented the postbg toggler, and it creates yet another problem, ah ah... If you infinite-scroll the page, nothing says the new page will start with the same, alternate postbg. Also, obviously, soft merging doesn't happen. I'll have to put this all into a new function, and call it at infinite time, like I do for breakLinks, IIRC...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Soft merging of posts
« Reply #5, on July 25th, 2013, 07:17 AM »
The thing about Stack Overflow and similar Q&A is that most people still manage to screw it up and post new messages rather than replying to things as they wanted to do, and the whole system gets screwed up as a result.

I'd be inclined not to have 'render just this layer' thing if possible... no need to get *too* complex with it. There is a level of functionality we need, and pretty much we have that. Anything else we'll add when we need it.

As for the alternation fixing, the solution to that would be to call the next page of results as not already-prerendered and then add it at post time, e.g. bring it from the server in JSON or a similar wrapper and have the wrapper be expanded to include classes in the client.

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Soft merging of posts
« Reply #6, on July 25th, 2013, 11:08 PM »
Quote from Arantor on July 25th, 2013, 07:17 AM
The thing about Stack Overflow and similar Q&A is that most people still manage to screw it up and post new messages rather than replying to things as they wanted to do, and the whole system gets screwed up as a result.
Never underestimate the average user's ability to screw up a perfectly logical interface ;)
Quote
I'd be inclined not to have 'render just this layer' thing if possible... no need to get *too* complex with it. There is a level of functionality we need, and pretty much we have that. Anything else we'll add when we need it.
I think that since it's basically a free addition (only adds a few instructions, once per skeleton rendition), it could be used for other things, for instance, creating a 'fake' mini-skeleton with multiple layers that follow each other, and then you simply call the layer to be rendered, multiple times, or just once. It could, err... "make sense", I guess.
Quote
As for the alternation fixing, the solution to that would be to call the next page of results as not already-prerendered and then add it at post time, e.g. bring it from the server in JSON or a similar wrapper and have the wrapper be expanded to include classes in the client.
Nah, because it wouldn't be possible to merge the posts with the previous page's.

Anyway, I've done it... Quite simply, I changed the infinite scroller to call a soft merge on all posts (including the old ones), every time a new series of posts is added. I simply had to add a .not() filter to ensure that sub-posts (inside a soft-merged post) weren't touched by the thing, because then it would return a user ID of 0 for these... (OTOH, I could simply tell Wedge to 'skip' an item if it has no user ID, but whatever, I think the filter will be faster, so let's do that...)

TL:DR, infinite scrolling + soft-merging is working now.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Soft merging of posts
« Reply #7, on July 26th, 2013, 05:14 AM »
That's precisely my point. SO has a simple enough interface and yet it's screwed up, especially by theoretically technically minded users. Now multiply that by the lowest common denominator of forum posters and boom. Even FB gets badly messed up by it with its very limited notions of it.

It's not so much the whole thing of being free vs not free, it's a user (mod author) usability issue. The more things people can do, the more complex troubleshooting becomes - bear in mind we will not only be supporting the core code but also supporting plugin authors trying to do things too.

Auk

  • Can I get a Box?
  • Posts: 64
Re: Soft merging of posts
« Reply #8, on July 26th, 2013, 09:13 AM »
I thought this feature got added a long time ago. I noticed it while reading the commits to this project.

With that said, I like where this is going. At least the threaded view of things, even though it might break the traditional forum topic appearance. A reply to a reply should appear as a new message. A threaded reply to a reply would kinda get funny to those not used to viewing threaded discussions. I've seen on some websites. Just doesn't appear too elegant.

I like how youtube does it. You can see all the messages,  and a reply to a reply are clasped, but you can still see their messages as a New message being replied to another commenter.

Nothing is more despicable than respect based on fear.

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Soft merging of posts
« Reply #9, on July 26th, 2013, 08:05 PM »
@Auk> I *especially* don't like YT's implementation of it... ;) Too confusing.
Quote from Arantor on July 26th, 2013, 05:14 AM
It's not so much the whole thing of being free vs not free, it's a user (mod author) usability issue. The more things people can do, the more complex troubleshooting becomes - bear in mind we will not only be supporting the core code but also supporting plugin authors trying to do things too.
Re: threading, I think that people are 'aware' of how it works, if anything billions of blogs (more or less........ ::)) are using Disqus for comments, for instance.

Anyway, I just discovered another issue... Argh.
"new" anchors don't get taken into account. So, I changed the HTML to output these into the postheader, and it would work, only... It seems that because the anchor is being moved by JS, Chrome no longer gives a damn about it, and it just won't lead me to any post, and as I said: argh... :-/

Well, I guess I'll have to take into account the possibility of doing it in ob_sessrewrite, right..?
Re: Soft merging of posts
« Reply #10, on July 26th, 2013, 11:05 PM »
I've reverted my anchor changes, so the 'new' links should be working again. (Except when the latest unread post is a double-post, like this one.)

I'm working, in parallel, on a PHP-only implementation of my JavaScript code.
I've decided that it should be much, much faster to do the changes on the buffer, if I use macros inside the msg skeleton, so this is exactly what I'm doing, and it has two advantages:
- easy to find individual messages, and thus merge them, if possible;
- since you can redefine macros in skins, I can (finally!) modify Wireless to remove some blocks of code entirely, instead of just hiding them. It's not a guarantee though, but at least I should be able to move all of the "if (MOBILE)" code away from PHP, and to macros, e.g. you can conditionally define a macro to be empty on 'mobile', or even more selectively, e.g. 'android[-3.9], ios[-5.2]', etc...

Isn't that exciting...? ;)
(What do you mean, "no it'll be even more complicated"..?!)

Still, I'm torn between keeping the macro definitions as they are, in index.template.php, or moving them all to skin.xml, just like I got rid of the skeleton definition in index to move it to skeleton.xml, but the latter case was done because it wasn't any different in terms of performance, while it'll be a bit slower in the case of a skin.xml macro definition.

I'll look into that tomorrow, I guess...
Re: Soft merging of posts
« Reply #11, on July 27th, 2013, 02:26 PM »
Quote
- since you can redefine macros in skins, I can (finally!) modify Wireless to remove some blocks of code entirely, instead of just hiding them.
Hmm, actually I just realized that I already went through this... Dunno why I forgot it, but it seemed like it was only recently, that Pete complained about Wireless still holding the HTML for most things, or something..?
There are multiple ways to remove code for a skin:
- hardcode the change into the template file (meaning it's not up to the skin, of course...)
- hardcode it into skeleton.xml (IIRC it supports either MOBILE_SKIN (i.e. the skin has a 'mobile' flag), or a browser string),
- hardcode it into the macros, like I'm doing now.
Quote
Still, I'm torn between keeping the macro definitions as they are, in index.template.php, or moving them all to skin.xml, just like I got rid of the skeleton definition in index to move it to skeleton.xml, but the latter case was done because it wasn't any different in terms of performance, while it'll be a bit slower in the case of a skin.xml macro definition.
I'll be going to the xml route, because I made some tests, and even with 60 custom macros, it takes no more than half a millisecond to parse them all, so it should be all right, no..? (Even if it isn't, I could just as well cache these for the next X minutes, etc.)

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Soft merging of posts
« Reply #12, on July 28th, 2013, 03:05 AM »
I really need to study the code to get a feel for this myself but from a plugin point of view, provided that adding new items to the userbox is just a case of using the existing skeleton and that adding stuff like menu buttons can still be done with the menu arrays (though, frankly, I'd kind of like internal helper functions because it's... complicated), it's probably all good.

live627

  • Should five per cent appear too small / Be thankful I don't take it all / 'Cause I'm the taxman, yeah I'm the taxman
  • Posts: 1,670
Re: Soft merging of posts
« Reply #13, on July 28th, 2013, 03:40 AM »
Quote from Arantor on July 28th, 2013, 03:05 AM
I really need to study the code to get a feel for this myself but from a plugin point of view, provided that adding new items to the userbox is just a case of using the existing skeleton and that adding stuff like menu buttons can still be done with the menu arrays (though, frankly, I'd kind of like internal helper functions because it's... complicated), it's probably all good.
it won''t work, because the call to the skeleton is in the template, and isn't grabbable through a hook.
A confident man keeps quiet.whereas a frightened man keeps talking, hiding his fear.

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Soft merging of posts
« Reply #14, on July 29th, 2013, 08:07 PM »
So... Took me longer than expected, but I did manage to rewrite Subs-Template to handle soft-merging of posts at the PHP level (ob_sessrewrite), and it works. You can check out the HTML source code, it's all done directly now.

Few caveats:
- #new still doesn't work for me, even though at this point it SHOULD... Maybe it's a browser bug..? :unsure:
- Parsing time takes about 10 milliseconds when there are many posts to merge. It's... quite a lot more than I expected. I dunno. It's still very good, and it saves bandwidth, after all...

Thoughts, anyone?