Wedge

Public area => The Pub => Features => Topic started by: Nao on October 31st, 2011, 11:08 PM

Title: Thought system
Post by: Nao on October 31st, 2011, 11:08 PM
Okay, I'll be opening this topic to discuss the upcoming thought system...

For those of you who aren't in our Friends group, you can't access thoughts in here, so I'll redirect you to noisen.com where you can see a list of thoughts on the homepage.

Basically, it's a chatbox, only it doesn't have all that Ajax stuff that makes it look like it's real time and mostly puts a lot more load on your server.

I'm about halfway through my implementation for Wedge. I rewrote a lot of it, and added some features which may or may not work.

- Privacy: determine whether you want your thoughts to be visible to everyone, just members, just your friends, or just you (e.g. a to-do...)
- Edit ANY of your thoughts, including older 'latest thought' or your replies to others' thoughts.
- The latest thought is now in the sidebar. You can either edit it, or write a new one (there are links for that.)
- If you edit a current thought, if the new version has less than 10% differing from the old thought, the original publishing date will be kept. Otherwise, the thought's publishing date is set to now. (This is different from the current implementation where a 20% difference will generate a new timestamp, also it allows you to edit anything.)
- I've removed the 'Default Personal Text' option which means nothing to me... And to anyone else, I'd venture into saying. (Hey, Pete isn't the only one entitled to deleting SMF crap :P)
- Users can no longer set their Personal Text in their profile, if only because they can now set it from any page in the sidebar...

The privacy stuff only has the UI implemented. The 'edit everything' code is barely started, so it doesn't work.
The JavaScript adds an extra couple of kilobytes to script.js, which bothers me a bit (though it'll probably be more like 500 bytes once minified an gzipped), but OTOH it's a pretty neat feature so I think it's best to have it as default...

I'm a bit stumped with storing the latest thought though. Right now I'm storing something like "1500|My thought" in the members table where 1500 is the thought's ID, so that I can easily tell Wedge whether we're currently editing the latest thought (by putting its ID into an invisible part of the form) or simply writing a new thought that looks a lot like an earlier one (in which case the ID is set to 0 at this point.)
I don't want (too much) to get rid of personal_text because some people may not want to use the thought system and still have their old SMF-imported personal text below their avatar and stuff. But I think that storing the thought ID as a 'hack' inside the personal_text itself is not exactly pretty.

Open to any suggestions... I'm off to bed, this weekend was hectic.
Title: Re: Thought system
Post by: Arantor on October 31st, 2011, 11:11 PM
I should add, it's a touch more than a chat box, it's actually closer to FB or Twitter - but only on your site for your members.

We have a surprising number of discussions that don't really warrant a topic of their own, and it's one of the reasons I post so little on FB or Twitter, is because mostly it has a relevance and context that it wouldn't have elsewhere.
Title: Re: Thought system
Post by: Nao on October 31st, 2011, 11:25 PM
I'm considering adding an option to send any of your public thoughts to twitter as well. If you are willing to enter your twitter password because I'm not willing to start playing with their API anyway :P
It's just that I never use twitter so it's no priority to me.

Did you understand my question on the field stuff? :^^;:
Title: Re: Thought system
Post by: Arantor on October 31st, 2011, 11:27 PM
Quote
I'm considering adding an option to send any of your public thoughts to twitter as well. If you are willing to enter your twitter password because I'm not willing to start playing with their API anyway
Um, I don't think you can do that any more, any time I've had a third party app of any description attempt it, it's taken me to Twitter to approve the application, which means it needs an auth key. If you're allowing user integration you must already have one anyway ;)

I did notice the implied question but I don't have a better answer short of storing the two details separately in the members table.
Title: Re: Thought system
Post by: Nao on October 31st, 2011, 11:32 PM
- you mean a website can't connect to them using a regular username and pw? You mean they willfully broke many sites?

- it's just that it's only a cache of sorts. If I store the Id I might as well loafd the thought from the thoughts table... Ah well.
Title: Re: Thought system
Post by: Arantor on October 31st, 2011, 11:34 PM
Quote
- you mean a website can't connect to them using a regular username and pw? You mean they willfully broke many sites?
Oh yes. It's amazing what happens you say 'XSS vulnerability' and 'account abuse' enough times. Every app that connects to Twitter does so using OAuth now.
Quote
- it's just that it's only a cache of sorts. If I store the Id I might as well loafd the thought from the thoughts table... Ah well.
Saves time, and it means another table not joined, thus another table not locked, less I/O, faster queries etc.
Title: Re: Thought system
Post by: Nao on November 1st, 2011, 05:17 AM
Xss like sniffing the pw over the network?
Anyway that makes it unlikely I'll work on twitter support. Although we should maybe add hooks to enable a tweet plugin to catch thoughts and tweet them.

Hmm as for personal text I'm still not sure what's best. Storing Id in a new field without a key, or having the hack thing and doing an explode on member load...
Title: Re: Thought system
Post by: Arantor on November 1st, 2011, 09:00 AM
XSS like abusing the crap out of you sending your username and password to spam Twitter. Which would be ironic if it weren't for the fact it was spamming malware links.

It would probably be better to have it as two separate columns for performance.
Title: Re: Thought system
Post by: Nao on November 1st, 2011, 10:08 AM
Kay then.

The member table is so huge, I'd just rather not add fields that aren't gonna be used by everyone.

Didnt we have plans for a generic joined sister table to remove weight from it btw?
Title: Re: Thought system
Post by: Arantor on November 1st, 2011, 10:25 AM
There's several fields in the members table already that you could argue about, e.g. pm prefs, notify prefs.

Splitting those off into a separate table might not be a bad idea, they're not required that often in reality.
Title: Re: Thought system
Post by: Nao on November 1st, 2011, 10:43 AM
The question would only be-- would it hurt performance if we joined that table on every member lookup?
Title: Re: Thought system
Post by: Arantor on November 1st, 2011, 10:49 AM
Yes, yes it would.

That's why it's important to figure out whether it's truly needed on every member lookup - or if it's not, when it is actually used.

For example, the pm prefs and notify prefs are only called for when handling PMs and notifications respectively, and putting them in a separate table probably isn't going to make a lot of difference (though, it gets complicated when figuring out the whole 'default setting' vs 'user setting' thing in that case without having an explicit row per user)

Whereas I can see the thought being pulled more often (like personal text is)
Title: Re: Thought system
Post by: Nao on November 2nd, 2011, 11:15 PM
WTF... The thought system is so complicated now ;)

Let's say I'm writing a new thought, and set it with a privacy value > 0, meaning 'not public'.
Wedge will record the ID and text in the member table, so that I can quickly get back my latest thought on each page load.

However, when it comes to showing the latest thought under your avatar in topic pages... How exactly is Wedge supposed to know whether it CAN show it to everyone? Does it mean storing the privacy value in the member table as well...? Or use some hack, like storing a negative value of the id_thought, meaning 'the absolute value of this is the actual id_thought, and I don't want anyone to see this thought in a post'?
Heck, I should probably have a potential privacy value saying 'public, but don't show under my avatar'... :^^;:

PS: id_thought is now stored in the member table as implied above. It's mainly because it's really used on every page...
Title: Re: Thought system
Post by: Arantor on November 2nd, 2011, 11:37 PM
Quote
However, when it comes to showing the latest thought under your avatar in topic pages... How exactly is Wedge supposed to know whether it CAN show it to everyone?
That's actually really easy, IMO.

Store the most recent, publicly visible one in the members table, and store the master record of everything in the table. You only need to retrieve the public one in topics, main profile etc., as everywhere else that uses thoughts will be doing so querying for privacy and thus will be using the main table anyway.

I like the idea of public/not under my profile. I'd probably use that all the time, actually - I personally find little or no benefit to having it in the post display as I have discussions etc. via the front page.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 07:47 AM
It wouldn't work that way. Thoughts are shown: in topic pages, in profile pages (not done yet), and on the homepage (possibly later an SSI function as well). Only the topic pages will use the stuff we mentioned. However, the main use for that field, for me, is in the sidebar where your current thought is being shown. If we only store the latest public thought (which BTW may sound silly to members who then can't see your member thoughts under your avatar), this will disrupt the sidebar block.

That's why it's complicated.

As for 'not under my avatar', currently the options (public/members/friends/self, all written as of now) are shown in a select box, but if I want to enable users to say they don't want their stuff in the topic pages, I'd need to do one of these: either have a profile option to completely disable sharing on topic pages (this would also be the logical destination of a 'share public thoughts on Twitter' option), or replace the select box with an Ajax popup called through a 'privacy' icon, where you actually have a list of checkboxes: 'show in topic pages', 'show to everyone', 'show to members only', 'show to my friends only'. If you disable everything, it's the equivalent of 'self' in the select box. If you enable 'show to everyone' and 'show to my friends only', you're just a cunt :P

I can't tell you the number of times I rewrote the thought system since last week... :lol: And there's still a lot of work to do. I'm feeling like everytime I want to tackle a small task like thoughts, I end up spending ten times longer on it than I first expected. The exact same thing happened for the skeleton system which I originally thought would take only a couple of days to implement... :^^;: (Well, it DID take only a couple of days. To get it working. But it was very basic then.)

Ahh, midnight! The Cinderella effect has caught up with me, and I'm out of pumpkins.
Posted: November 3rd, 2011, 12:01 AM

(bump!)

Maybe I should commit my work in progress for now?
(It does not work well and I'd rather not.)
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 10:58 AM
Oh, I see, you always need the latest, public or not, for any given user - even if you're going to be serving the public one inside topic views.

There's no good way around that. Either you have to store both in the members table, or you have to store them both in the main thoughts table and query for them appropriately. Or one of each.
Quote
I'm feeling like everytime I want to tackle a small task like thoughts, I end up spending ten times longer on it than I first expected.
I know exactly what you mean :/
Posted: November 3rd, 2011, 10:58 AM

As for progress, commit what you're happy with.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 11:44 AM
Quote from Arantor on November 3rd, 2011, 10:58 AM
There's no good way around that. Either you have to store both in the members table, or you have to store them both in the main thoughts table and query for them appropriately. Or one of each.
Well, I'm thinking we may have to do a join on the display page... I don't like the idea, but... Well, I don't know. I'd like more opinions.
It's this, or we get three fields in the members table: personal_text, thought and id_thought... :(
Quote
I know exactly what you mean :/
That plugin feature is a bitch eh?
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 11:50 AM
Quote
Well, I'm thinking we may have to do a join on the display page... I don't like the idea, but... Well, I don't know. I'd like more opinions.
Technically, it's in loadMemberData where the join is. Hmm, lMD needs more branches than just minimal, normal, profile. Didn't we already talk about that before? :/
Quote
That plugin feature is a bitch eh?
The plugin management itself is pretty straightforward. It's all the crap that goes with it, like dealing with file permissions[1] and figuring out how the update process should work.
 1. Seriously, while I understand the principles and so on of the way Linux/Unix file permissions are done, the whole lot of having to use FTP to elevate permissions on most systems instead of allowing the PHP process to run as the right user is just so mindnumbingly broken...
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 12:46 PM
Okay...... I *think* I've found our solution, or something like that.
We add a special text data field in the member table, which will hold a serialized array of *all* variables that are only of use to the member in question.

id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Things like smiley_set or id_theme/skin could also be in it (although the latter would prevent easily giving stats in the Change Skin page.)
Maybe timezone and language stuff, too. (It's used in a user's profile, though. But we could have some kind of function that gets the serialized array and extracts some data from it. As long as it's not used for more than *two* users at a time (the viewer and the viewee), it's probably safe.)
message_labels maybe... But it's a text field so it could end up eating the new data field's space. We could make the data field a LONGTEXT but it could end up wasting a lot of time unserializing data that's never used (outside the PM area.)
instant_messages and unread_messages are good candidates, too.

Well... What do you think? I for one would sure enjoy seeing a stripped down member table!
And TE's importer can easily deal with serializing imported fields.

Oh, while looking at the member table, I found a couple of definitions ending with default 0 instead of default '0'... I turned them to '0' but I've always wondered the point of using a string when an integer is perfectly fine... Maybe it was for compatibility with SQLite or PGSQL or something...? Maybe we should just go ahead and use integers....
Posted: November 3rd, 2011, 12:44 PM
Quote from Arantor on November 3rd, 2011, 11:50 AM
Technically, it's in loadMemberData where the join is. Hmm, lMD needs more branches than just minimal, normal, profile. Didn't we already talk about that before? :/
Yes we did. And like with so many conversations, I guess nobody thought of adding this to their to-do list... :P (Mind you, it's probably the same on SMF's side.)
Quote
The plugin management itself is pretty straightforward. It's all the crap that goes with it, like dealing with file permissions[1] and figuring out how the update process should work.
 1. Seriously, while I understand the principles and so on of the way Linux/Unix file permissions are done, the whole lot of having to use FTP to elevate permissions on most systems instead of allowing the PHP process to run as the right user is just so mindnumbingly broken...
I completely agree with your view on FTP, but there's really nothing we can do.
I'd tend to say, "to hell with people who use strange filesystems", but the more we say that kind of thing, the less users we'll have... And after spending 14 months on the project, I don't want it to fall into oblivion, even though I didn't care about that on day one.
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 01:12 PM
Quote
id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Works for me.
Quote
Things like smiley_set or id_theme/skin could also be in it (although the latter would prevent easily giving stats in the Change Skin page.)
I'm not fussed about smiley set,[1] though theme/skin would probably be better to keep out for reporting purposes.
Quote
Maybe timezone and language stuff, too. (It's used in a user's profile, though. But we could have some kind of function that gets the serialized array and extracts some data from it. As long as it's not used for more than *two* users at a time (the viewer and the viewee), it's probably safe.)
Timezone doesn't bother me, unless we want to do some kind of display of user timezones. Language, however, has other consequences, like when sending email templates, it actually tries to fetch the right one for the language of the user the email is going to.
Quote
message_labels maybe... But it's a text field so it could end up eating the new data field's space. We could make the data field a LONGTEXT but it could end up wasting a lot of time unserializing data that's never used (outside the PM area.)
Offhand, I'm not sure where message_labels is evaluated in terms of how it figures out how to label a PM (given that it's essentially a rules filter)
Quote
instant_messages and unread_messages are good candidates, too.
Not they're not. If you send someone a PM, do you really want to have to query their member rows, unserialize each one, increment, then update each row? If you send 10 people a PM, that's 11 queries. Now imagine sending a newsletter to each member via PM. As opposed to a single query that updates all of those things at once (which means only one table lock)
Quote
Well... What do you think? I for one would sure enjoy seeing a stripped down member table!
And TE's importer can easily deal with serializing imported fields.
Keeping the table lean is important, provided you don't end up compromising other things like above.
Quote
Oh, while looking at the member table, I found a couple of definitions ending with default 0 instead of default '0'... I turned them to '0' but I've always wondered the point of using a string when an integer is perfectly fine... Maybe it was for compatibility with SQLite or PGSQL or something...? Maybe we should just go ahead and use integers....
The dump is from either mysqldump or phpMyAdmin (not sure which) which escapes everything, anything that's without quotes is one added manually somewhere.

It wasn't compatibility, SQLite and PGSQL got their own install SQL scripts.
Quote
Yes we did. And like with so many conversations, I guess nobody thought of adding this to their to-do list...
On top of that we were talking about using wesqlQuery with it too, which is probably why no-one did it, because it's hellishly complicated (and important to get right)
Quote
I completely agree with your view on FTP, but there's really nothing we can do.
I'd tend to say, "to hell with people who use strange filesystems", but the more we say that kind of thing, the less users we'll have...
It depends on your definition of strange. From my point of view, the majority of filesystems are strange :P I did consider avoiding the whole upload process entirely and forcing users to use FTP but that's not very nice, now, is it?
 1. Honestly, how many sites actually do the whole multi-smiley-set thing? Most people neither know nor care. But I don't have enough reason to remove that.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 03:04 PM
Looking at some of your code in updateMemberData...

Code: [Select]
if ((!is_array($members) && $members === $user_info['id']) || (is_array($members) && count($members) == 1 && in_array($user_info['id'], $members)))

Couldn't this be simplified to this?

Code: [Select]
if ((array) $members === (array) $user_info['id'])

Untested... Just wondering ;) It works if I use echo (array) '1' === array('1') for instance.
Quote from Arantor on November 3rd, 2011, 01:12 PM
Quote
id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Works for me.
I take it you agree that a data field is a good idea then :)
Quote
Timezone doesn't bother me, unless we want to do some kind of display of user timezones. Language, however, has other consequences, like when sending email templates, it actually tries to fetch the right one for the language of the user the email is going to.
Okay, scrapping this one then...
Quote
Not they're not. If you send someone a PM, do you really want to have to query their member rows, unserialize each one, increment, then update each row? If you send 10 people a PM, that's 11 queries. Now imagine sending a newsletter to each member via PM. As opposed to a single query that updates all of those things at once (which means only one table lock)
I didn't know that instant_messages was calculated when sending a PM. I thought it was done when going to your PM area or something... So, scrapped.
unread_messages is probably in a similar situation.

Clearly, though, I'll be doing the data field because it's a good opportunity for plugins to store profile-specific data without having to add their own database fields (or custom fields).
Quote
The dump is from either mysqldump or phpMyAdmin (not sure which) which escapes everything, anything that's without quotes is one added manually somewhere.
So there's really no reason to keep it that way... :)
I always wondered about that but didn't want to break it.
Quote
On top of that we were talking about using wesqlQuery with it too, which is probably why no-one did it, because it's hellishly complicated (and important to get right)
Ah, yes, that was it... I remember now how I was frustrated having to cancel my upcoming commit about that... :lol:
Quote
It depends on your definition of strange. From my point of view, the majority of filesystems are strange :P I did consider avoiding the whole upload process entirely and forcing users to use FTP but that's not very nice, now, is it?
We could also simply say -- if your upload process doesn't work, use FTP and don't bother us with that! Just outputting a nice error message explaining what happened is okay to me...
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 03:17 PM
Quote
Looking at some of your code in updateMemberData...
That was one of mine? Egad, looks ugly. Smells more like something that was there before and that I changed.
Quote
I take it you agree that a data field is a good idea then
I'm a fan of such things, and I'm going to be making use of such when I get to implementing the per-user sanctions (e.g. through warnings and having permissions removed etc.)
Quote
I didn't know that instant_messages was calculated when sending a PM. I thought it was done when going to your PM area or something... So, scrapped.
unread_messages is probably in a similar situation.
Yeah, it's done at point of sending and updated when you read messages, the idea being to minimise the effort required in gathering the data for the purposes of updating something that appears on every page. No sense requerying when you can just update it at the point you're physically increasing the number of items that it relates to. The downside is that it is possible to get it out of whack, though it isn't that common.
Quote
Clearly, though, I'll be doing the data field because it's a good opportunity for plugins to store profile-specific data without having to add their own database fields (or custom fields).
Well... it sort of depends. If it's going to be a mostly passive thing like timezone, it's cool to do. But anything that is primarily a form of cache (like unread_messages) shouldn't be put there. Of course, there's no reason why you couldn't use the themes table if you're careful to store per-user settings anyway (since it's already abused for the custom profile fields)

Let me put it another way. Right now, one of the things I came to regret in WedgeDesk was the way the number of tickets is cached and not stored in the profile of the user. The overhead, despite caching, for that is surprising - and it has other consequences too that I won't get into.[1] So for the next version I'm going to be putting it into the members table.

While at first blush it would make sense to put it into the cache you're proposing or into the themes table, the best solution for me is a new column in the members table. From my perspective, the value is obtained immediately (though as I discussed elsewhere, I don't really like the fact that $user_settings is persisted and contains the *entire* row, regardless of what $user_info has), and I can update it when necessary regardless for users with a single UPDATE, which I can't do in the case of either the cache (because I then have to do one update per user) or the themes table (because I have to establish the value exists to UPDATE it or do an INSERT first)

It's useful for data that's per user but only accessed and updated by that user, or at least, very infrequently by other users.
Quote
I always wondered about that but didn't want to break it.
At least, that's my understanding of it. I can't imagine any other reason.
Quote
Ah, yes, that was it... I remember now how I was frustrated having to cancel my upcoming commit about that...
Quote
We could also simply say -- if your upload process doesn't work, use FTP and don't bother us with that! Just outputting a nice error message explaining what happened is okay to me...
Which means for a fraction of users, they have a cushy ride and everyone else is told to hit the highway. To be honest, I don't feel that's particularly acceptable, it's basically saying "I couldn't be bothered to make this work well for you." in my head.
 1. Suffice to say, SSI has fun with it, and the logic around cache purging when tickets are updated is ridiculous.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 03:52 PM
Quote from Arantor on November 3rd, 2011, 03:17 PM
Quote
Looking at some of your code in updateMemberData...
That was one of mine? Egad, looks ugly. Smells more like something that was there before and that I changed.
Ah, my bad! I just checked in SMF 2, and it's also there, so you have nothing to do with it ;)
I assumed it was your code because it's hook code, and I didn't remember writing it, and SMF has so little hook code that I just didn't even consider they could have written it.

I should have known better :lol:
Quote
I'm a fan of such things, and I'm going to be making use of such when I get to implementing the per-user sanctions (e.g. through warnings and having permissions removed etc.)
Good to know!
Any issues with the field being called 'data', btw...? That's what it is called in my local install. It's not very original and can be confusing, but at least it makes for a nice $user_info['data'] array.
Quote
Let me put it another way. Right now, one of the things I came to regret in WedgeDesk was the way the number of tickets is cached and not stored in the profile of the user. The overhead, despite caching, for that is surprising - and it has other consequences too that I won't get into.[1]
 1. Suffice to say, SSI has fun with it, and the logic around cache purging when tickets are updated is ridiculous.
Well there's definitely some overhead too for the AeMe cache and it's stored in the profile table...
Quote
and I can update it when necessary regardless for users with a single UPDATE,
Yeah, that's the thing with the data field: it's only for when you need to update one member at a time... Which is why the custom function I added for it (updateMyData(), can be renamed later) doesn't take a $members param -- it uses $user_info['id'] directly, at least for now.
Quote
Which means for a fraction of users, they have a cushy ride and everyone else is told to hit the highway. To be honest, I don't feel that's particularly acceptable, it's basically saying "I couldn't be bothered to make this work well for you." in my head.
Do you have any stats, honestly.....? :-/
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 04:07 PM
Quote
Any issues with the field being called 'data', btw...? That's what it is called in my local install. It's not very original and can be confusing, but at least it makes for a nice $user_info['data'] array.
'data' is a little generic, but it works.
Quote
Well there's definitely some overhead too for the AeMe cache and it's stored in the profile table...
There's overhead whatever you do, the only way not to have overhead is to not do anything to the menu in the first place.[1]

But in WD's case, the overhead is actually more trivial; it's not just looking up last-read-id vs last-actual-id or similar, there's a whole bunch of determination of ticket status vs author vs permissions to figure out how many tickets should be shown if the user is staff, and how many if they're a regular user. I spent far too long fighting it and it would save so much effort to just put it in the members table where it can be updated as needed rather than queried and regenerated at best every few minutes (and at worst far more often)

Horses for courses. Do what works best for what you're doing, and in my case I realised too late that I should have put it in the members table.
Quote
Yeah, that's the thing with the data field: it's only for when you need to update one member at a time... Which is why the custom function I added for it (updateMyData(), can be renamed later) doesn't take a $members param -- it uses $user_info['id'] directly, at least for now.
Yup, and that's cool - provided that you are only updating one member at a time. The data field makes that a requirement, not a contextual thing. (You can update multiple rows to SET column = column + 1 in a single query, but you can't update multiple rows based on multiple criteria)
Quote
Do you have any stats, honestly.....?
http://www.simplemachines.org/community/index.php?board=9.0 and http://www.simplemachines.org/community/index.php?board=147.0 searching for 'ftp' and 'chmod'. Those are the users who can't do it out of the box. Pretty much all crap shared hosting (which to be fair, is almost all of it) runs the web server as www-data or nobody or apache, which means it's not writable.

And while there is an argument to be made about using chmod, the idea of making the Plugins folder a+rw doesn't fill me with confidence, especially because it then becomes an attack vector (especially on shared hosts) - it's not like the attachments or avatars folders, where you're expressly locking down content from being executed, the contents of Plugins and its related folders absolutely *must* remain executable. Which means elevating privileges, uploading, then dropping privileges again.
 1. The only winning move is not to play.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 11:07 PM
Quote from Arantor on November 3rd, 2011, 04:07 PM
There's overhead whatever you do, the only way not to have overhead is to not do anything to the menu in the first place.[1]
 1. The only winning move is not to play.
<ahhhhhhhhh, that movie...>
Quote
Yup, and that's cool - provided that you are only updating one member at a time. The data field makes that a requirement, not a contextual thing. (You can update multiple rows to SET column = column + 1 in a single query, but you can't update multiple rows based on multiple criteria)
Sure.

As for the data field... I'm looking into the members table and it's pretty hard finding something that isn't used in a regular query.
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.

passwd_flood may be a candidate. I'd like to invite you to look into it... Try a global search for it. In some situations, it's retrieved from the database, but never used later. Look inside Reminder.php for instance. My guess is that it was planned to be used in validatePassword() (again, I'd suggest looking through all occurrences of validatepassword.) This may be an actual SMF bug...

secret_answer/question is a strong candidate too. It's used in a regular query, but it's always LIMIT 1 so it's worth doing an unserialize on it every time. Same problem as pm_prefs though, for saving.

Anything to justify having this new field, really... :lol:
Quote
Pretty much all crap shared hosting (which to be fair, is almost all of it) runs the web server as www-data or nobody or apache, which means it's not writable.
Even if it's running as www-data etc, I believe there are some ways to configure the server to make it work...

I use MANY hosts in the past. Only one had issues with modifying the filesystem through PHP, IIRC.

PS: feel free to fix any bugs you find in the thought system, now that it's committed. 'Modify' doesn't work anymore, I'll look into it tomorrow.
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 11:11 PM
Quote
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.
Yeah, that one's pretty mindblowing to understand.
Quote
passwd_flood may be a candidate. I'd like to invite you to look into it... Try a global search for it. In some situations, it's retrieved from the database, but never used later. Look inside Reminder.php for instance. My guess is that it was planned to be used in validatePassword() (again, I'd suggest looking through all occurrences of validatepassword.) This may be an actual SMF bug...
I will look at that, I never knew what the field did anyway.
Quote
secret_answer/question is a strong candidate too. It's used in a regular query, but it's always LIMIT 1 so it's worth doing an unserialize on it every time. Same problem as pm_prefs though, for saving.
It's created in one place, it's only used in one place, I'm not seeing what the problem is?
Quote
Even if it's running as www-data etc, I believe there are some ways to configure the server to make it work...
Yes... making files writable by default. Making an area of your space, that's going to be executable by live request, open to be modified by anyone else on the server.
Title: Re: Thought system
Post by: Nao on November 3rd, 2011, 11:19 PM
You probably missed the PS I added after my post :)
Quote from Arantor on November 3rd, 2011, 11:11 PM
Quote
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.
Yeah, that one's pretty mindblowing to understand.
The list? It's probably a good example of how a good idea (gathering similar data saving into a single array for easier maintenance) gets blown out of proportion and becomes totally unmaintainable because of silly validation functions etc. It's documentation in the comments is even more confusing. As if it was all written to be maintained only by its original writer...
Quote
I will look at that, I never knew what the field did anyway.
Good opportunity ;)
Quote
It's created in one place, it's only used in one place, I'm not seeing what the problem is?
No problem, except for the saving of it.
Quote
Yes... making files writable by default. Making an area of your space, that's going to be executable by live request, open to be modified by anyone else on the server.
Hmm alright...

Okay, going to bed early tonight. I'm tired... I think I've never spent so much time changelogging a feature. I must have re-re-re-reviewed the entire file list about 10 times in a few days...
Title: Re: Thought system
Post by: Arantor on November 3rd, 2011, 11:36 PM
Quote
You probably missed the PS I added after my post
I did, yes.
Quote
The list? It's probably a good example of how a good idea (gathering similar data saving into a single array for easier maintenance) gets blown out of proportion and becomes totally unmaintainable because of silly validation functions etc. It's documentation in the comments is even more confusing. As if it was all written to be maintained only by its original writer...
It's a true SMF variable then.
Quote
Hmm alright...
It's not for nothing that vB and xF store their templates in the database...
Quote
Okay, going to bed early tonight. I'm tired... I think I've never spent so much time changelogging a feature. I must have re-re-re-reviewed the entire file list about 10 times in a few days...
Sleep well :)
Title: Re: Thought system
Post by: Nao on November 6th, 2011, 12:11 PM
Just wanted to say that it took one more week than expected, but I'm hoping to have the thought system implemented by tonight... Or tomorrow if I'm too tired (only got 5 hours sleep... -_-)

So it's basically what we have on Wedge.org, except that:
- you can edit your current thought or create a new one (on wedge.org, if you want to edit your thought you need to edit it word by word or the site will mistake it for a new thought),
- it has privacy settings (no changes compared to the current SVN though...),
- you can reply to any thought (need to consider whether they should inherit their parent's privacy setting... there's no quote so it's not a BIG deal for now), but you can also delete any of your thoughts (or any thought if you're a global moderator), and edit your own replies.

The code is relatively ugly for now... I had to make concessions and I hope I will be able to get rid of them later.
Also, you can no longer simply click a thought to answer it. You have to click the Reply button next to it -- not because it's impossible, but because it makes more sense to click a thought to *edit* it, and because I don't want to make the mistake of editing a third party's thought instead of replying it (as the admin), I selfishly made it impossible to click directly...
Title: Re: Thought system
Post by: Arantor on November 6th, 2011, 02:12 PM
There's a side benefit of that for touch users, if someone posts a link as a thought, it means it's accessible directly as opposed to tapping once to activate edit and subsequently disable the event handler, so that the link becomes accessible directly.

I think the choice of forcing a discrete button press is not a bad one, you still have the notion of a button press anyway but it's much clearer as to what you're doing :)
Title: Re: Thought system
Post by: Nao on November 6th, 2011, 07:12 PM
That's one of the things I hate about the current thought system on Wedge.org, yeah... ;)

I added a css transition on showing the buttons but it's probably a tad too much... Once I commit, try removing the 'transition' line at the end of index.member.css and tell me what you like best.

Also, I'm doing .php_htmlspecialchars() and .php_unhtmlspecialchars() as required, but I'm not exactly a specialist of wedit::preparsecode(), so would you mind looking into it later...? Just to tell if it's best to use it or not...

95% finished.
Title: Re: Thought system
Post by: Nao on November 6th, 2011, 07:30 PM
Also, there's an issue with my current logic... I'm preparsing the BBC before storing it, but then I allow for editing posts, which means that if they have BBC, they end up showing HTML tags in the code. Hmm...
I guess I could use the HTML to BBC converter, especially since Wedge only accepts a small subset of very simple bbc tags in thoughts.

I'll probably commit everything before I get to it. I spent many hours stuck in the Parisian traffic, I'm definitely KO tonight. I want to commit and be done with it... Whatever glitches it has.
Title: Re: Thought system
Post by: Arantor on November 6th, 2011, 07:37 PM
I would not perform htmlspecialchars on the client side. I would leave it as is, and do it on the server side. Always assume the client is insecure/unsafe.

Strictly speaking, anything that's going to be sent, no matter how indirectly, to the bbc parser 'should' go through preparsecode, and if memory serves, preparsecode does not perform htmlspecialchars anyway, because I seem to recall it's always done ahead of time by other processes.

It's complicated because there are other side matters to attend to with it. preparsecode actually does an insane amount of stuff, IIRC the list in no particular order is:

* convert newlines to br tags (not entirely sure why it does this, I've never fully understood this logic but I'm assuming it's for the benefit of code tags so that you don't try second guessing p tags, and it means if you want a double blank line, you get one)
* validates that anyone using the html bbcode is an admin (or remove the tag otherwise)
* breaks the content of nobbc bbcode so that it won't be parsed (physically convert the brackets to their numeric equivalent)
* attempt to fix table tag hierarchies through regular expressions (table must contain tr, tr must contain td, td must be inside tr, tr must be inside table, list must contain li, li must be inside list)
* convert double space to nbsp+space so they're actually displayed as such
* break instances of [] and [' so that they're not parsed as potential bbcode for speed (e.g. inside people posting code outside of code tags)
* convert /me to the me bbcode
* convert specific colour bbcodes to the generic case (e.g. replace red, blue, etc, bbcode with color=red, color=blue)
* invoke the tag scheme checker to go through img, url and similar tags looking for valid http/s entries and forcing them to have the right scheme
* if images are posted and there is a maximum image size, validate the image's size and rewrite the img bbcode if necessary to include new sizes

It is pretty insane when you think about it! But that's what's required to ensure that any arbitrary posting is safe to be displayed later. (More, in fact, could be done, because it could step through the tags at save time and validate that they are properly nested and so on, and take that burden off parse-bbc)

Long story short, it depends on what you're looking to allow in the thought system. If it's the small list of inline codes, you don't need to invoke the entire preparser, but it will need a subset of the loop at the end of the routine where it splits the post into sections by code tags, most importantly to catch the fixTags call (to sanitise links and/or images posted, though images should probably be rewritten to links anyway!)

You don't need to have it also do the regexps that handle fixing table/list nesting because they're not going to be parsed anyway, so no need to alter them (and I'd suggest not altering the thought for that purpose)

Perhaps there should be a parameter to preparsecode, for inline parsing requirements, where if false, only do what's necessary for inline bbc requirements.
Quote
which means that if they have BBC, they end up showing HTML tags in the code. Hmm...
This is a problem that has affected many, many people in figuring this stuff out. The HTML to BBC converter is not rugged, but if you're only feeding it the inline tags, it's probably enough.
Title: Re: Thought system
Post by: Nao on November 6th, 2011, 10:46 PM
'kay, haven't added preparsecode() to the stuff in the end... This is mainly because I didn't even remember correctly how I was doing things, uh... The only thing that got parsed at cache time was personal_text, while all other free thoughts are parsed at display time. I still think there's an issue with bbcode though... <sigh>

One thing that I should point out is that I was using .escape() to send the data in JS, I forgot that it didn't support UTF, so now it uses (properly) .php_urlencode().
Title: Re: Thought system
Post by: Arantor on November 6th, 2011, 11:00 PM
Quote
I still think there's an issue with bbcode though...
Which is probably why preparsecode is needed. I'll take a look in a bit and see what I think is needed.
Title: Re: Thought system
Post by: Nao on November 6th, 2011, 11:06 PM
Take your time, I'll be offline in a few minutes anyway.
Title: Re: Thought system
Post by: Nao on November 7th, 2011, 08:37 PM
BTW... I woke up :P

Also, there are a few things I mentioned in the changelog topic...
For instance-- how about we rename the system to 'Words'?
It may be less intuitive, though. Because it doesn't start with T(weet) but with W(edge).
I just thought it'd be worth asking...

Friggin' system is taking more time than I want, BTW. I'm currently working on a 'proper' profile template... Long and annoying to test.
Title: Re: Thought system
Post by: Arantor on November 7th, 2011, 08:43 PM
Unless we borrow from Facebook and call it a status...
Title: Re: Thought system
Post by: Dismal Shadow on November 7th, 2011, 08:43 PM
How about 'protest'? :P
Title: Re: Thought system
Post by: Arantor on November 7th, 2011, 08:49 PM
Quote from ~DS~ on November 7th, 2011, 08:43 PM
How about 'protest'? :P
OccupyInternet
Title: Re: Thought system
Post by: Nao on November 7th, 2011, 10:50 PM
Quote from Arantor on November 7th, 2011, 08:43 PM
Unless we borrow from Facebook and call it a status...
That would imply non-comment thoughts have more weight than comments... I'm not sure I want to do that..?

Anyone?
Title: Re: Thought system
Post by: Dismal Shadow on November 7th, 2011, 10:56 PM
Quote from Arantor on November 7th, 2011, 08:49 PM
Quote from ~DS~ on November 7th, 2011, 08:43 PM
How about 'protest'? :P
OccupyInternet
OccupyTime?
Title: Re: Thought system
Post by: Arantor on November 7th, 2011, 11:00 PM
Haha

@Nao, you're right, of course, it would elevate their apparent standing. Hmm, I got nothing better. :(
Title: Re: Thought system
Post by: Nao on November 7th, 2011, 11:16 PM
I'm assuming 'Words' sounds bad in English, then...?
Title: Re: Thought system
Post by: Dismal Shadow on November 7th, 2011, 11:20 PM
Quote from Nao on November 7th, 2011, 11:16 PM
I'm assuming 'Words' sounds bad in English, then...?
Yeah, kinda. Doesn't appeal to me to be honest.
Title: Re: Thought system
Post by: Arantor on November 7th, 2011, 11:21 PM
Quote from Nao on November 7th, 2011, 11:16 PM
I'm assuming 'Words' sounds bad in English, then...?
It's just bland. Thoughts at least is more descriptive.
Title: Re: Thought system
Post by: Nao on November 7th, 2011, 11:56 PM
It's just less pronounceable to me.
But that's okay! I just figured that if someone found a better word for Thoughts, they should feel free to participate :)
Posted: November 7th, 2011, 11:37 PM

(Pete, could you also tell me your 'thoughts' on the current Wedge implementation of thoughts..? Is it okay for you? I know I'm still missing showing off the Edit/Reply/Delete buttons because ahem there's no 'latest thoughts' list on the homepage for now. Only on my install. Will commit something tomorrow in that respect.)
Title: Re: Thought system
Post by: Arantor on November 8th, 2011, 12:05 AM
I've just been so wrapped up with other stuff I haven't even looked at it yet :(
Title: Re: Thought system
Post by: Dismal Shadow on November 8th, 2011, 12:27 AM
Quote from Arantor on November 7th, 2011, 08:43 PM
Unless we borrow from Google+ and call it a stream...
Fixed it for you. :P
Title: Re: Thought system
Post by: godboko71 on November 8th, 2011, 12:40 AM
Insight or pondering... anything is better then words :/

Edited to add: I like "Thoughts." The name makes one pause and think before they post... wonder what Facebook/Twitter would be like if its own wording made on think for a split second.
Title: Re: Thought system
Post by: Nao on November 8th, 2011, 10:12 PM
Is it worth splitting the Thought object into its own file...?
It is 3.09KB for now, and possibly more in the future.

Pros:
- this saves about 1KB (after minification and gzipping) for guests (9.48KB -> 8.65KB)

Cons:
- adds a '-thoughts' or something to the script URL (not a big deal)
- adds extra file_existes/filemtime etc on each page load
- why move only one object to its own file, and not all? (only, all other objects are called by guests too...)

So at this point... I'm not exactly sure what's best.
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 02:17 PM
Bump.

Also... What's left to integrate into Wedge before I can install it on Wedge.org...?

I'm not sure at this point. I'll keep a copy of the database (and site) around in a sub-folder after making the switch, so I can easily import anything missing, but I'd like to be sure I have as much available as possible beforehand.
Title: Re: Thought system
Post by: Arantor on November 9th, 2011, 03:31 PM
Blog support might need tidying up, but that's it really. Just need to remember board access UI doesn't work if more boards are created.
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 03:42 PM
Yep, yep...

My, is it me or are we just in a quiet period? :P

I'm hurrying up to finish Risen (which I long left taking dust in a shelf if I may say), before I get to start Skyrim... Plus I'm starting to hate taking two weeks to implement something I originally intended to do in two days... :whistle:
Title: Re: Thought system
Post by: TE on November 9th, 2011, 04:20 PM
Quote from Nao on November 9th, 2011, 02:17 PM
Also... What's left to integrate into Wedge before I can install it on Wedge.org...?
IMO at least merging attachments / avatars / media items, or was this dropped?
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 05:53 PM
No, it isn't dropped... It's just scary. The amount of work involved in this... Considering it took me 7 more time to implement thoughts than expected, and I expect the attachment/avatar rewrite to take a week at best, that would make it 7 weeks if I'm optimistic... :^^;:
Thing is, we *can* perfectly use Wedge here without doing the rewrite, because once the feature is written, I'll write an internal converter to turn regular attachments into AeMe items. Or we'll just include it into your importer... (IIRC one can do a partial import of only some specific data... Well maybe I can import the current database's attachment table into the current database's media table :P)

All media items at wedge.org at considered 'temporary' anyway. They may or may not be deleted later.
Title: Re: Thought system
Post by: Aaron on November 9th, 2011, 07:18 PM
I can see why you think of it as a scary undertaking, Nao, really.

To be quite honest, I'm still not entirely sold on the idea of dropping the current attachment system in favour of a system completely integrated with the media gallery.

Here's a crazy idea: make Wedge an attachment-less system by default, and let webmasters choose between two plugins: the current attachment system (i.e. SMF's) or the Aeva Media Gallery.

Probably a crazy idea, but wanted to post it regardless. ;)
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 07:37 PM
Would mean current installs would be unable to use AeMe..?
Title: Re: Thought system
Post by: Arantor on November 9th, 2011, 08:09 PM
I know why it's complex, but if you turn it round a bit, it makes sense.

The core attachment system has two principle weaknesses, in my book: firstly, it's not extensible, and secondly its management capabilities are limited.

Really, all an attachment is is a file uploaded to the system... Post attachments are restricted by the topic permissions, album attachments by the album permissions. The more you look, the clearer it is that they have an awful lot in common, probably more than is realised.

And I'm not really seeing the problems with doing it, other than the fear of doing it. All the issues related to how I think it can be made extensible aren't really issues at all, I already covered off how that would likely work, at least in my head, by giving each item a source to have originated from (both a place and a contextual id, such as album+1 for an item in album 1 or help desk+2 for an item attached to a helpdesk ticket, etc.)
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 08:53 PM
Which is hardly indexable?
Title: Re: Thought system
Post by: Arantor on November 9th, 2011, 09:50 PM
It doesn't need much indexing, it's mostly accessed by primary key anyway...
Title: Re: Thought system
Post by: Nao on November 9th, 2011, 11:23 PM
Hmm, then...
Title: Re: Thought system
Post by: Arantor on November 9th, 2011, 11:50 PM
Media items are always referred to by their id only, which is a PK lookup, and then a lookup against the album's permissions.

Attachments are currently done as a request referring to the attachment id, plus the topic id. I'm not quite sure why that's done that way, because it causes two problems. Firstly, it means that if a message is moved to another topic, any pre-existing link in posts is broken. Secondly, it means there's a lookup of attachments table (by attach id) to messages table (by id_msg) which will then return the topic.

The only reason the topic id is even given in the first place is so that ;topic= is filled in the URL and the board permissions are implicitly loaded as a by-product. That means a BFQ[1] is issued, one that queries the topics table, the boards table, the categories table, plus fetching the board's owner, board moderators... yeah. That's not clever.

So pushing it solely to id and managing the access and permissions internally would almost certainly be at least as fast, if not faster, than the current approach.

The exact method to be used is up for debate, since it implies allowing for unexpected alterations and extensions later, but the principle seems clear enough. That's the 'main' problem with using attachments, from my perspective.

The other issue is relating to how the file path is determined. Right now, the path is determined based on the album's information. What I figure is that, invariably, a second query is going to have to be made. The first query identifies that the file exists, and would return the information necessary to locate and pass to the authorisation subsystem.

The authorisation subsystem would not only identify that the file can be passed on to the user but should also return the path to it.

As far as files being attached to posts, we could retain the current UI. Once the files are uploaded, we can insert the equivalent of the smg bbcode into the post itself, meaning that users are free to reposition the items within the page if they wish.

I'm sure there's more but that's all I have right now in my little brain for it. (Busy watching The 7th Voyage of Sinbad. Awesome film.)
 1. Big fucking query.
Title: Re: Thought system
Post by: Arantor on November 10th, 2011, 10:33 AM
Getting back to the original topic, I find two problems and one thing I don't understand.

First up, there's a bug in it whereby nobbc is not handled properly. It's not a potential security issue as far as I can see, because htmlspecialchars is run still, so even though nobbc is not parsed the same way as other tags, it won't do anything bad. It just won't work as expected. (Since nobbc is available to the inline parser, we need to run preparsecode on it to actually carry out the inlining on it, I did wonder if that would be the case, though.)

Second, the JS is broken if Packer is used. I haven't figured out quite where that is yet, but it gets very upset with "Unexpected token (" which breaks all remaining JS on the page (which is problematic as it also breaks things like the menus, fortunately menu navigation isn't the only route to Server Settings now ;))

Also, is it intentional that links posted in thoughts should not be clickable? In fact... the URL tag doesn't work. It's parsed in the display view but it's raw BBC in both the thoughts view in profile and in the sidebar box.

The one thing I don't entirely get is how a user is supposed to reply to thoughts, there's no interface like in Noisen or here.

Otherwise though, it looks pretty good :)
Posted: November 10th, 2011, 10:29 AM

On a related note, the inline parser has 'color', 'black', 'blue', 'green', 'red', 'white' listed, but this should just be 'color' as preparsecode does the conversion. (Though it does it case-sensitively.)
Title: Re: Thought system
Post by: Nao on November 10th, 2011, 11:55 AM
- I'll test with Packer.
- I haven't been using preparsecode, as I said before. It's something I'll have to deal with, too.
- Sidebar box doesn't show a parsed thought because the Ajax stuff doesn't return anything. It was done this way on noisen, never cared to develop further... Obviously, I *should* return a parsed string to show. And I will.
- BTW, if you add a URL between nb tags, it won't be parsed into a url tag. Just a limitation I noticed yesterday in here... Dunno if it's in Wedge too, but I suppose so.
- I'm kind of on a break right now because, well... I thought I was done but I started logging a lot of items I should do, and I figured, oh crap this is going to take me another week... So right now, it's all you get :P

- Color stuff: ah, yes, thanks for the tip...
Posted: November 10th, 2011, 11:51 AM

Update: not getting any problems with Packer...?
Title: Re: Thought system
Post by: Arantor on November 10th, 2011, 12:02 PM
Quote
- I haven't been using preparsecode, as I said before. It's something I'll have to deal with, too.
I'd assume that newlines should be stripped before sending to preparsecode, but that's the only gotcha as far as I'm concerned.
Quote
- Sidebar box doesn't show a parsed thought because the Ajax stuff doesn't return anything. It was done this way on noisen, never cared to develop further... Obviously, I *should* return a parsed string to show. And I will.
I had a horrible discovery on that note the other day. I put a thought reply of 'I made him say <odd item> though... ' in and I was more than a shade bothered when I found it didn't protect the tag on the inline save (but of course it is when it's held on the backend, and reloading the page is of course fine)
Quote
- BTW, if you add a URL between nb tags, it won't be parsed into a url tag. Just a limitation I noticed yesterday in here... Dunno if it's in Wedge too, but I suppose so.
Really? Fairly sure I've put links in footnotes before and it worked as expected. In other news I've been quietly wondering if the footnotes couldn't be made even more awesome: if you have two footnotes with the same contents, I feel that only one entry should be put at the bottom and the same number used in both cases. The main reason this came into my head was because for one thing I want to do, I want to be able to do Wikipedia like citations by reusing footnotes instead of having to roll my own (because I have a project in mind that I would rather use a customised Wedge, or even a customised SMF,[1] for than MediaWiki)
Quote
I'm kind of on a break right now because, well... I thought I was done but I started logging a lot of items I should do, and I figured, oh crap this is going to take me another week... So right now, it's all you get
I know the feeling. I'm currently tackling moving the calendar into its own plugin. It's... interesting.
Quote
- Color stuff: ah, yes, thanks for the tip...
Side note: I seem to recall the bbcode parser will chew through uppercase bbcode e.g. [WHITE] but tags like that won't be converted, because the regular expression just replaces white|blue|black|red|green with color=$1 without any callback or inline eval to strtolower it. (Since this is at preparsecode time rather than display time there's no reason why it couldn't be rewritten to use a case-insensitive match and strtolower it once found)
Quote
Update: not getting any problems with Packer...?
Chrome doesn't like it for some reason. This was off a totally fresh install, too.
 1. As much as developing on Wedge is awesome, for a project that is a one-off codebase, I'd rather build it once and leave it without having to do regular patching and updates, if that makes sense.
Title: Re: Thought system
Post by: AngelinaBelle on November 15th, 2011, 08:36 PM
I second the whole moved-topic-attachment-link problem -- learned not to link to attachments. Only to gallery items.

Using a gallery for attachments is complicated because then the attachment can have a permission COMPLETELY different from the topic/board it is attached to, which could be tricky and confusing.

Using the gallery instead of attachments. But that's OT.
Title: Re: Thought system
Post by: Arantor on November 15th, 2011, 09:44 PM
That's precisely the point: while normally a gallery item is tied to the album it was in, I figure that if you attach an item to a post, it should use the permissions of that post, but there's no reason it has to advertise it to the user, it should "just work", and we can display a footnote somewhere on the item to indicate who can see it if needed.
Title: Re: Thought system
Post by: Nao on November 16th, 2011, 01:07 AM
But it's the eternal question: does the user WANT their item to be tied to the post's permissions...? Maybe they want it to be accessible from the gallery.
Which is (probably) why we need to enable this only for items in an 'attachments' album, i.e. if the user chooses to post in another album, the item will use the album's permissions, while if they post in the general attachment album, it'll use the post's permissions and a reprint won't be allowed in another post.

Which of course also begs the question... Are people going to understand how it all works :lol:
Title: Re: Thought system
Post by: Arantor on November 16th, 2011, 01:17 AM
Quote
Which is (probably) why we need to enable this only for items in an 'attachments' album, i.e. if the user chooses to post in another album, the item will use the album's permissions, while if they post in the general attachment album, it'll use the post's permissions and a reprint won't be allowed in another post.
That wasn't really my intention, because it's not really want I would want to use. I'd personally prefer that it uses the permissions but that reposting should be permitted, so that if I attach a file to a post, that isn't in a regular album, I would want to be able to repost it elsewhere - for example, the logo madness thread here.

If it's tied to visibility behind the scenes so that users don't have to worry about it, they can just repost the image without any concern, e.g. through whatever equivalent of the smg tag we end up with. That way, it's no longer tied to the *topic* as it currently is (with little good reason, IMO)

The idea, primarily, is to make it as easy as possible to get files onto a post with as little effort as practical - it being in the gallery makes it easier to reuse later.
Title: Re: Thought system
Post by: DoctorMalboro on November 16th, 2011, 02:39 AM
I like the idea, but I don't know how many people will use it with other social networks added, maybe it could be added as a plugin the social integration...
Title: Re: Thought system
Post by: Arantor on November 16th, 2011, 02:41 AM
I somehow don't think it's going to become a plugin.

It was a requirement for migrating this site to Wedge (because it's actively used here) but it might end up being a path for reposting to, say, Twitter from the forum.
Title: Re: Thought system
Post by: Nao on November 16th, 2011, 10:07 AM
Quote from Arantor on November 16th, 2011, 01:17 AM
That wasn't really my intention, because it's not really want I would want to use. I'd personally prefer that it uses the permissions but that reposting should be permitted, so that if I attach a file to a post, that isn't in a regular album, I would want to be able to repost it elsewhere - for example, the logo madness thread here.
That seems to be in line with what I suggested..? What exactly is there to clear up between our visions of the thing?

Also, I'm starting to wonder if we shouldn't simply keep the attachments table and files and everything, and *just* move most of the front-end and back-end manipulation to the Aeva codebase...
Well, that would be something in-between no implementation and full implementation. I don't particularly like that either...
Quote
If it's tied to visibility behind the scenes so that users don't have to worry about it, they can just repost the image without any concern, e.g. through whatever equivalent of the smg tag we end up with.
(That's the media tag, now, BTW.)

Also, regarding rev 1165... I must admit I haven't followed any developments regarding this bug. I had a quick look this morning, and apparently it's a subject that's been discussed at the AeMe topic at sm.org these last few days, with a weird fix posted, and I'm not even sure what your fix does, whatever, over at sm.org it seems the problems are with Firefox 8 and here they're with IE9...?!
Title: Re: Thought system
Post by: Arantor on November 16th, 2011, 12:53 PM
Quote
That seems to be in line with what I suggested..? What exactly is there to clear up between our visions of the thing?
You said that if it were tied to posts, users wouldn't be able to repost it... whereas specifically I'm saying they should be able to do so. They just don't need to worry about the other details.
Quote
Also, regarding rev 1165... I must admit I haven't followed any developments regarding this bug. I had a quick look this morning, and apparently it's a subject that's been discussed at the AeMe topic at sm.org these last few days, with a weird fix posted, and I'm not even sure what your fix does, whatever, over at sm.org it seems the problems are with Firefox 8 and here they're with IE9...?!
OK, here's the problem.

There's the Content-Disposition header, which is supposed to indicate what the browser is supposed to do with the content that follows, except the specification pretty much states it should be ASCII characters only. Which is fine, right up until you have high bit characters in the name, which in non English forums is more common than you'd think. So there is a quasi-extension that's now pretty much standardised which allows a content provider to state that the data is really UTF-8, except browser support is quirky for it.

The original code in SMF issues separate different headers for IE, Firefox and Opera, I changed that a year ago to IE and everyone else.

Now, the problem is that IE9 got its act together and doesn't like the header being used for IE because it's special, ugly and it wants to conform with the in-crowd. Trouble is, Wedge recognises IE9 as IE (and SMF does not), so it was given the IE specific codepath which spits out the wrong filename on IE9. So the first change was to make the IE-only code path into IE8 and down.

The second change is that FF8 stopped accepting the otherwise-standard header, it would ignore the filename entirely because it didn't like the quote in the name. So the quote has to be changed for another delimiter, which caused it to start choking on spaces and other things, so the entire thing has to be urlencoded and it must also include spaces in that to be %20, rather than +, hence the rawurlencode call.

As far as I can tell, this works properly on all the big browsers. I'm not so sure it would work so well for SMF even though I gave the few lines in question to be reposted; there was a rather curt "this is the same fix I posted" note, except that it isn't, for several reasons (my fix doesn't bother to differentiate ISO and UTF-8, and I don't have a special case for FF, merely IE8 and down - when they get round to listing IE9, they'll have to fix it again, but hey, what do I know?)
Title: Re: Thought system
Post by: Nao on November 16th, 2011, 11:14 PM
Thanks for the details.
Just for the records, what I did in rev 1167 is trying to follow the test cases described here:
http://greenbytes.de/tech/tc2231/
A real goldmine... And scary, too! Anyway -- hopefully I got the code right.
Title: Re: Thought system
Post by: Arantor on November 16th, 2011, 11:19 PM
I'll take a look in a bit; Safari was the one main system I hadn't tested it on.

I opined elsewhere that part of the problem with SMF's fixes to this problem is that they didn't have a battery of tests to work on, meaning that they were fumbling around in the dark trying to find directions. In the same discussion, I noted that the list of tests to be carried out is:
Quote
* A filename with no special characters or symbols other than file delimiter (file.txt)
* A filename with 7-bit symbols characters, but no spaces and no + (file_test.txt, file-test.txt)
* A filename with 7 bit characters, and a space (file test.txt)
* A filename with 7 bit characters, and a + (file+test.txt)
* A filename with 7 bit characters, and a space and a + (file test+results.txt)
* A filename with high bit characters (any ISO encoding, on ISO encoded forum) (file™.txt)
* A filename with high bit characters and a space (ISO encoding) (file name™.txt)
* A filename with high bit characters and + (ISO) (file+test™.txt)
* A filename with high bit characters and a space and + (ISO) (file test+results™.txt)

All of the high bit character tests will then need to be repeated on a UTF-8 forum with UTF-8 encoded filenames, and if any of the 7 bit tests fail, there is no point with the high bit tests until the 7 bit tests pass.
In fact I realise I missed single and double quotes off the list but " is not valid in most systems' filenames anyway. (And in the code I wrote, it would be safely rawurlencode'd anyway)
Title: Re: Thought system
Post by: Nao on December 12th, 2011, 06:03 PM
Oh my... It would appear the little f*cker is dead.

I don't know why or how. It just doesn't register when I edit or type in a new thought.

Can anyone who tested the thought system (on a local copy) so far give me some feedback on this...? Did you ALWAYS see that issue? Send a thought -> the return copy is empty and nothing is entered into the DB. No errors either...
I really don't want to bother with Ajax debugging right now :(
Posted: December 12th, 2011, 06:01 PM

Oh, and I also messed up rev 1179... It was last month already -- and I committed a file that wasn't supposed to be (index.css), with selectbox styling for a feature that wasn't even in...

Pete or anyone, when you see that kind of file and it's NOT documented, can you report it please...? It usually means I forgot to uncheck a file that's not ready for consumption... (And yes I do have a fair share of files I have to uncheck every time. Pretty annoying really. But if I start putting them on Ignore, I'll just forget to deal with them in the future...)
Title: Re: Thought system
Post by: Arantor on December 12th, 2011, 06:06 PM
Quote
Oh my... It would appear the little f*cker is dead.

I don't know why or how. It just doesn't register when I edit or type in a new thought.
Hmm, I thought this was fixed? It's worked for me though I haven't tested it recently.
Quote
Pete or anyone, when you see that kind of file and it's NOT documented, can you report it please...?
See, I didn't realise it wasn't supposed to be in there, knowing that the selectbox was coming up...
Title: Re: Thought system
Post by: Nao on December 12th, 2011, 07:39 PM
- It was fixed and working great, at least when I last dealt with it... i.e. it must have broken after that, while fixing something else in an unrelated place. Can you confirm and look into it?

- Just to be clear: if I ever commit "work in progress" code with shitty commented out stuff, it is NOT as planned, unless I'm clearly specifying it in the changelog (I usually apologize for this kind of thing.) I also found out that rev 1186 has my work-in-progress script.js committed... Just the same: it's shitty and not as planned. And again, it's not mentioned in the changelog.

Every time you commit something, I'll make sure to ensure you haven't committed any 'weird' stuff that's not in the changelog, and if there is, I'll ask you about it... I can do that kind of mistake, too, so I was kinda hoping someone was checking on my commits...
Title: Re: Thought system
Post by: Nao on December 13th, 2011, 03:25 PM
Found the culprit... In Ajax.php, rev 1184, I added a wedit::preparsecode($text) on the thought. Uh...

So... It doesn't generate an error in the database. Hard to trace. I simply removed the line from my current rev and it works. And it's not related to my fixNesting stuff because it wasn't already there in rev 1184... :-/
Title: Re: Thought system
Post by: Nao on December 15th, 2011, 10:39 AM
(Fixed.)
Title: Re: Thought system
Post by: TE on February 15th, 2012, 08:31 AM
question: is there an option to enable / disable the thoughts system inside the admin area? Maybe I'm blind but I can't fine one   :angel:
Title: Re: Thought system
Post by: Nao on February 16th, 2012, 03:49 PM
Nope... It's obviously something I'll add later on. But it's not my priority, honestly. I want to give it a chance to get known... :P (Obviously, at the very least it'll be enabled by default...)
Title: Re: Thought system
Post by: Nao on May 24th, 2012, 11:32 PM
Okay, so it's not ready for use on Wedge.org but I decided to throw away the thought list on profile pages -- never intended to keep this because it created huge pages and was a bit buggy.

So, here's how it'll be done now:

- The homepage lists the latest thoughts, like now.
- The author link will point to their actual profile, not their profile page's thought list
- There's a new mini-icon next to the thought date. Follow it to reach the thought thread page.

The thought thread is basically all thoughts gathered under the same id_master, i.e. any thought that sprang as a reply to a single thought.
It can create a page with one thought, or a page with several dozens -- although it's very unlikely.

The profile pages will now list only the author's thoughts -- without any contextual thoughts around them, split by pages, and accompanied by a link to the corresponding thought thread.

Although it makes it a bit 'harder' to follow someone's thoughts in context, I think it's definitely the better way to show thought lists. Really... (Plus, it makes the code a bit simpler.)

PS: I also fixed several minor bugs today, in case you didn't notice on wedge.org, eheh...
PPS: that quick edit animation looks really neat... I think it'll take more than a week for me to get bored of it :P
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 06:56 PM
It certainly covers viewing a single conversation, which is the ideal IMO.

What you seem to lose is an easy way to view everything off a single person but I'm not sure that's a great loss at this time. I think it's going to be something we need to let run and let people play with and get to use. That said, I have a feeling your instinct is right on this one, it's usually pretty good.
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 07:47 PM
Actually, Wedge.org still shows thoughts by a single user...? It's in the same place as always: profile > show thoughts.
The difference now is that they're shown just like on the main page. There's a link to the thread for each thought, as well as a "(...)" in case there are replies to a particular thought. (I know, it may not be very intuitive...)
I still need to add paging to these, though...

Re: what I said on the skin topic, how are we going to manage 'walls'?
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 07:54 PM
/meis tired, was not sure (hence the 'what you seem to lose...' comment)
Managing walls in what respect?
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 08:19 PM
I think I'm done with the thought rewrite... Phew.
Not perfect, but good enough.

Walls ==> well, at one point we're going to offer user streams, right...? We already discussed doing them like some of the admin logs, to help with translations. So, what about the thought system? Do we use thoughts for the streams, or do we copy any thoughts into the stream, assuming that a stream may have a shorter or longer 'expiry date' than thoughts...?
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 08:23 PM
Oh, I see what you mean.

Since it's not a wall, per se, but a stream of notifications (at least that's what I want to do), you only need to worry about notifying replies up the chain, and really that's only necessary up to the immediate parent IMO.

The stream by definition has a shorter life (I'm thinking 3-5 days for notifications you've seen before, no limit on unread notifications, for now at least)
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 08:34 PM
Quote from Arantor on May 25th, 2012, 08:23 PM
Since it's not a wall, per se, but a stream of notifications (at least that's what I want to do), you only need to worry about notifying replies up the chain, and really that's only necessary up to the immediate parent IMO.
Eh..?
Quote
The stream by definition has a shorter life (I'm thinking 3-5 days for notifications you've seen before, no limit on unread notifications, for now at least)
So, I take it you don't want to do something like FB, right..?

And what about wall comments? The thought system is well suited for that, too... (Of course, we could also consider these to be part of the 'profile' type of board, and create a new topic on each notification being commented... Uh... Then that means multiple topics per user...? How do we store that... Meta field eh?)
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 08:48 PM
I always thought of it being more like Facebook notifications than a full 'wall' per se.
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 09:57 PM
Wall works better for me...
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 10:18 PM
So are we building a forum or a social network?
Title: Re: Thought system
Post by: godboko71 on May 25th, 2012, 10:34 PM
Wall and Notifications are different things in my mind.
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 10:37 PM
They are, but they're not. The underlying structure is virtually identical.

I'm in favour of having notifications a la Facebook and XenForo but I'm not a fan of walls per se.
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 11:01 PM
A notification goes away when you view it. I like having a slightly longer archive...
Title: Re: Thought system
Post by: Nao on May 25th, 2012, 11:39 PM
Plus. If you have 100 friends, does that mean any of my actions get entered 100 times into the db...?
I'd rather have it done once. I can control things more clearly. Facebook does both but between the two I prefer the wall system. Seems more realistic.

I'd still like to find a way to make it possible to comment streams without having to create a new table for these comments. Basically thoughts were there to also encourage posting small messages without fear of filling the main messages table too much. Ensuring better performance in the long run.

Well. I guess it's something I'll have to decide by myself. But having a 'stream' type for boards seems a bit like overkill. Does this mean creating a new topic on each new wall entry...? then we might as well give up thoughts. Or give up posting thoughts into the stream. Hmm maybe having a stream and a thought list in parallel isn't something unrealistic or unreliable.
Title: Re: Thought system
Post by: Arantor on May 25th, 2012, 11:45 PM
That's the point, the notifications system I'm talking about doesn't go away straight away - it lives for a few days.

You wouldn't normally notify all your friends on an action. You would, however, normally notify them if it's a thread they're watching and you reply to it. Or like it etc.

This is also the thing about having a wall - it magnifies a lot of things, in a way that a notifications system doesn't; notifications are essentially personal while a wall is everyone's, and I don't particularly want to build a social network (even if a forum is a specific type of that)
Title: Re: Thought system
Post by: live627 on May 26th, 2012, 12:56 AM
Front page thought list seems to have more padding. Looks much better!
Title: Re: Thought system
Post by: Nao on May 26th, 2012, 08:10 AM
I know! Eheh.
I don't know why it had padding issues recently. Probably due to the skin loading order changes.
Title: Re: Thought system
Post by: Nao on May 26th, 2012, 08:55 AM
Quote from Arantor on May 25th, 2012, 11:45 PM
That's the point, the notifications system I'm talking about doesn't go away straight away - it lives for a few days.
So, what if your friends never go online more than once a month...? They'll never see more than 10% of your notifications?
I'd tend to say, "people should be able to subscribe to a custom web/xml feed that takes whatever data they want from whatever account they want...", and that shouldn't be linked to a specific wall. (It'd just be, somehow... "choose what kind of data you want to see in your aggregated wall.")

It still doesn't solve the issues of where we store this data, whether we use the thoughts table, and where we store the comments :^^;:
Quote
You wouldn't normally notify all your friends on an action. You would, however, normally notify them if it's a thread they're watching and you reply to it. Or like it etc.
Hmm yes, I see what you mean. Although I do get FB notifications for stupid things like new posts in a group I'm on... (Okay, to be fair, FB allows me to disable these on a group by group basis.)
Quote
This is also the thing about having a wall - it magnifies a lot of things, in a way that a notifications system doesn't; notifications are essentially personal while a wall is everyone's, and I don't particularly want to build a social network (even if a forum is a specific type of that)
It is a social network, whether you want it or not... It's up to admins to determine how 'social' they want to be, I'd say.
Title: Re: Thought system
Post by: Arantor on May 26th, 2012, 02:16 PM
Quote
So, what if your friends never go online more than once a month...? They'll never see more than 10% of your notifications?
That wasn't what I said ;) *Read* notifications disappear after a few days, *unread* ones never disappear. Though there's an argument about the DB cluttering up. *shrug* This is what XenForo does and it's considered pretty useful. (Having actually prodded at it, I'm increasingly inclined to agree about its usefulness)
Quote
Hmm yes, I see what you mean. Although I do get FB notifications for stupid things like new posts in a group I'm on... (Okay, to be fair, FB allows me to disable these on a group by group basis.)
Yup.
Quote
It is a social network, whether you want it or not... It's up to admins to determine how 'social' they want to be, I'd say.
Of course it is, but you know what I mean.

We're arguing about two different things at this point. I'd rather keep thoughts on their own and have the 'wall' figured out as needed if it's going to be included, but honestly it's not something I'm even remotely interested in (building another Facebook does not interest me at all and to me seems to distract from the goal of Wedge as a killer forum), but the notifications side I'm much more interested in.
Title: Re: Thought system
Post by: Dragooon on May 27th, 2012, 01:42 PM
Sorry if I missed something in the discussion, but honestly Wall and Notifications are 2 very different things. Wall allows to subscribe to the public activity of the user, whereas notifications can be used by plugins etc. as well in order to notify the user (something which doesn't make sense as a wall feature). As a plugin author I'd be delighted to have a notification system which allows a more intermediate form of communication for things which are not important enough for things that'd warrant a PM or E-Mail (I got a couple of ideas of plugins that can use a notification system).
Title: Re: Thought system
Post by: Arantor on May 27th, 2012, 03:18 PM
I realised they were very different things but to me that could have a common root, namely related to who can see what.

FWIW, I'm not interested in a wall, but a notifications system would be *very* useful. I would prefer to see a wall as a plugin.
Title: Re: Thought system
Post by: godboko71 on May 29th, 2012, 01:02 PM
Notifications could have some neat uses if a user is active (online) they could get a notification about a reply or like to a topic instead of an email. Or if it's a site I goto often I might rather have no mail from the site but would love notifications that a thought or thread was replied to or liked. Say someone made an arcade plug in that plugin could send you a notification for a new high score in a game you had the high score or new games in subscribed genres.

If your friends with someone could get notifications for new thoughts and topics too.

I would not hate having a wall feed though as an admin and user I would be more interested in the notifications.
Title: Re: Thought system
Post by: Nao on May 29th, 2012, 01:31 PM
Alrighty.
Title: Re: Thought system
Post by: godboko71 on May 29th, 2012, 01:38 PM
Quote from Nao on May 29th, 2012, 01:31 PM
Alrighty.
I do like the idea of a agragated feed or wall. Subscribed boards could show new topics in your feed, thoughts posted on your wall could be shown friends thoughts could be shown. On sites with walls inhaled some notifications could be shifted to the wall instead, friends new thoughts for example would then make more sense in the feed then in the notification center.
Title: Re: Thought system
Post by: Nao on June 8th, 2012, 10:26 AM
So. Was considering thought likes.

There are two ways of implementing them.

The first one would be Pete's field. Using the likes system. Quite obvious.
The second one would be mine. Closer to retweets in look and feel. Adding a Rethink button. When you do that, it actually sends an *empty* reply. Internally this would be considered like a RT. Any reply to that RT would have their parent set to be the original thought. This allows getting an idea of a thiught's popularity by number of replies only.

What do you like best..?
Title: Re: Thought system
Post by: Arantor on June 8th, 2012, 03:37 PM
I think I'd prefer rethinks. Seems more logical and probably faster.
Title: Re: Thought system
Post by: Nao on June 8th, 2012, 04:53 PM
Certainly faster. And not too hard to implement I think :P
Title: Re: Thought system
Post by: Dragooon on June 8th, 2012, 04:54 PM
Certainly the second one.
Title: Re: Thought system
Post by: Nao on June 8th, 2012, 04:56 PM
I'm just not sure about the homepage list... See, what do I show for a new thought that's just a rethought?

"RT original"?

Or

"@original> RT"?

Or do I keep it aside and then add, next to the original thought, a list of people who rethought it?
Or do I just ignore that and test against thought != ''?

Lol it's getting fun even before I got started... Heck, even before I got finished with my other work :lol: