Thought system

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #15, on November 3rd, 2011, 10:58 AM »
Oh, I see, you always need the latest, public or not, for any given user - even if you're going to be serving the public one inside topic views.

There's no good way around that. Either you have to store both in the members table, or you have to store them both in the main thoughts table and query for them appropriately. Or one of each.
Quote
I'm feeling like everytime I want to tackle a small task like thoughts, I end up spending ten times longer on it than I first expected.
I know exactly what you mean :/
Posted: November 3rd, 2011, 10:58 AM

As for progress, commit what you're happy with.
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #16, on November 3rd, 2011, 11:44 AM »
Quote from Arantor on November 3rd, 2011, 10:58 AM
There's no good way around that. Either you have to store both in the members table, or you have to store them both in the main thoughts table and query for them appropriately. Or one of each.
Well, I'm thinking we may have to do a join on the display page... I don't like the idea, but... Well, I don't know. I'd like more opinions.
It's this, or we get three fields in the members table: personal_text, thought and id_thought... :(
Quote
I know exactly what you mean :/
That plugin feature is a bitch eh?

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #17, on November 3rd, 2011, 11:50 AM »
Quote
Well, I'm thinking we may have to do a join on the display page... I don't like the idea, but... Well, I don't know. I'd like more opinions.
Technically, it's in loadMemberData where the join is. Hmm, lMD needs more branches than just minimal, normal, profile. Didn't we already talk about that before? :/
Quote
That plugin feature is a bitch eh?
The plugin management itself is pretty straightforward. It's all the crap that goes with it, like dealing with file permissions[1] and figuring out how the update process should work.
 1. Seriously, while I understand the principles and so on of the way Linux/Unix file permissions are done, the whole lot of having to use FTP to elevate permissions on most systems instead of allowing the PHP process to run as the right user is just so mindnumbingly broken...

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #18, on November 3rd, 2011, 12:46 PM »
Okay...... I *think* I've found our solution, or something like that.
We add a special text data field in the member table, which will hold a serialized array of *all* variables that are only of use to the member in question.

id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Things like smiley_set or id_theme/skin could also be in it (although the latter would prevent easily giving stats in the Change Skin page.)
Maybe timezone and language stuff, too. (It's used in a user's profile, though. But we could have some kind of function that gets the serialized array and extracts some data from it. As long as it's not used for more than *two* users at a time (the viewer and the viewee), it's probably safe.)
message_labels maybe... But it's a text field so it could end up eating the new data field's space. We could make the data field a LONGTEXT but it could end up wasting a lot of time unserializing data that's never used (outside the PM area.)
instant_messages and unread_messages are good candidates, too.

Well... What do you think? I for one would sure enjoy seeing a stripped down member table!
And TE's importer can easily deal with serializing imported fields.

Oh, while looking at the member table, I found a couple of definitions ending with default 0 instead of default '0'... I turned them to '0' but I've always wondered the point of using a string when an integer is perfectly fine... Maybe it was for compatibility with SQLite or PGSQL or something...? Maybe we should just go ahead and use integers....
Posted: November 3rd, 2011, 12:44 PM
Quote from Arantor on November 3rd, 2011, 11:50 AM
Technically, it's in loadMemberData where the join is. Hmm, lMD needs more branches than just minimal, normal, profile. Didn't we already talk about that before? :/
Yes we did. And like with so many conversations, I guess nobody thought of adding this to their to-do list... :P (Mind you, it's probably the same on SMF's side.)
Quote
The plugin management itself is pretty straightforward. It's all the crap that goes with it, like dealing with file permissions[1] and figuring out how the update process should work.
 1. Seriously, while I understand the principles and so on of the way Linux/Unix file permissions are done, the whole lot of having to use FTP to elevate permissions on most systems instead of allowing the PHP process to run as the right user is just so mindnumbingly broken...
I completely agree with your view on FTP, but there's really nothing we can do.
I'd tend to say, "to hell with people who use strange filesystems", but the more we say that kind of thing, the less users we'll have... And after spending 14 months on the project, I don't want it to fall into oblivion, even though I didn't care about that on day one.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #19, on November 3rd, 2011, 01:12 PM »
Quote
id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Works for me.
Quote
Things like smiley_set or id_theme/skin could also be in it (although the latter would prevent easily giving stats in the Change Skin page.)
I'm not fussed about smiley set,[1] though theme/skin would probably be better to keep out for reporting purposes.
Quote
Maybe timezone and language stuff, too. (It's used in a user's profile, though. But we could have some kind of function that gets the serialized array and extracts some data from it. As long as it's not used for more than *two* users at a time (the viewer and the viewee), it's probably safe.)
Timezone doesn't bother me, unless we want to do some kind of display of user timezones. Language, however, has other consequences, like when sending email templates, it actually tries to fetch the right one for the language of the user the email is going to.
Quote
message_labels maybe... But it's a text field so it could end up eating the new data field's space. We could make the data field a LONGTEXT but it could end up wasting a lot of time unserializing data that's never used (outside the PM area.)
Offhand, I'm not sure where message_labels is evaluated in terms of how it figures out how to label a PM (given that it's essentially a rules filter)
Quote
instant_messages and unread_messages are good candidates, too.
Not they're not. If you send someone a PM, do you really want to have to query their member rows, unserialize each one, increment, then update each row? If you send 10 people a PM, that's 11 queries. Now imagine sending a newsletter to each member via PM. As opposed to a single query that updates all of those things at once (which means only one table lock)
Quote
Well... What do you think? I for one would sure enjoy seeing a stripped down member table!
And TE's importer can easily deal with serializing imported fields.
Keeping the table lean is important, provided you don't end up compromising other things like above.
Quote
Oh, while looking at the member table, I found a couple of definitions ending with default 0 instead of default '0'... I turned them to '0' but I've always wondered the point of using a string when an integer is perfectly fine... Maybe it was for compatibility with SQLite or PGSQL or something...? Maybe we should just go ahead and use integers....
The dump is from either mysqldump or phpMyAdmin (not sure which) which escapes everything, anything that's without quotes is one added manually somewhere.

It wasn't compatibility, SQLite and PGSQL got their own install SQL scripts.
Quote
Yes we did. And like with so many conversations, I guess nobody thought of adding this to their to-do list...
On top of that we were talking about using wesqlQuery with it too, which is probably why no-one did it, because it's hellishly complicated (and important to get right)
Quote
I completely agree with your view on FTP, but there's really nothing we can do.
I'd tend to say, "to hell with people who use strange filesystems", but the more we say that kind of thing, the less users we'll have...
It depends on your definition of strange. From my point of view, the majority of filesystems are strange :P I did consider avoiding the whole upload process entirely and forcing users to use FTP but that's not very nice, now, is it?
 1. Honestly, how many sites actually do the whole multi-smiley-set thing? Most people neither know nor care. But I don't have enough reason to remove that.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #20, on November 3rd, 2011, 03:04 PM »
Looking at some of your code in updateMemberData...

Code: [Select]
if ((!is_array($members) && $members === $user_info['id']) || (is_array($members) && count($members) == 1 && in_array($user_info['id'], $members)))

Couldn't this be simplified to this?

Code: [Select]
if ((array) $members === (array) $user_info['id'])

Untested... Just wondering ;) It works if I use echo (array) '1' === array('1') for instance.
Quote from Arantor on November 3rd, 2011, 01:12 PM
Quote
id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Works for me.
I take it you agree that a data field is a good idea then :)
Quote
Timezone doesn't bother me, unless we want to do some kind of display of user timezones. Language, however, has other consequences, like when sending email templates, it actually tries to fetch the right one for the language of the user the email is going to.
Okay, scrapping this one then...
Quote
Not they're not. If you send someone a PM, do you really want to have to query their member rows, unserialize each one, increment, then update each row? If you send 10 people a PM, that's 11 queries. Now imagine sending a newsletter to each member via PM. As opposed to a single query that updates all of those things at once (which means only one table lock)
I didn't know that instant_messages was calculated when sending a PM. I thought it was done when going to your PM area or something... So, scrapped.
unread_messages is probably in a similar situation.

Clearly, though, I'll be doing the data field because it's a good opportunity for plugins to store profile-specific data without having to add their own database fields (or custom fields).
Quote
The dump is from either mysqldump or phpMyAdmin (not sure which) which escapes everything, anything that's without quotes is one added manually somewhere.
So there's really no reason to keep it that way... :)
I always wondered about that but didn't want to break it.
Quote
On top of that we were talking about using wesqlQuery with it too, which is probably why no-one did it, because it's hellishly complicated (and important to get right)
Ah, yes, that was it... I remember now how I was frustrated having to cancel my upcoming commit about that... :lol:
Quote
It depends on your definition of strange. From my point of view, the majority of filesystems are strange :P I did consider avoiding the whole upload process entirely and forcing users to use FTP but that's not very nice, now, is it?
We could also simply say -- if your upload process doesn't work, use FTP and don't bother us with that! Just outputting a nice error message explaining what happened is okay to me...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #21, on November 3rd, 2011, 03:17 PM »
Quote
Looking at some of your code in updateMemberData...
That was one of mine? Egad, looks ugly. Smells more like something that was there before and that I changed.
Quote
I take it you agree that a data field is a good idea then
I'm a fan of such things, and I'm going to be making use of such when I get to implementing the per-user sanctions (e.g. through warnings and having permissions removed etc.)
Quote
I didn't know that instant_messages was calculated when sending a PM. I thought it was done when going to your PM area or something... So, scrapped.
unread_messages is probably in a similar situation.
Yeah, it's done at point of sending and updated when you read messages, the idea being to minimise the effort required in gathering the data for the purposes of updating something that appears on every page. No sense requerying when you can just update it at the point you're physically increasing the number of items that it relates to. The downside is that it is possible to get it out of whack, though it isn't that common.
Quote
Clearly, though, I'll be doing the data field because it's a good opportunity for plugins to store profile-specific data without having to add their own database fields (or custom fields).
Well... it sort of depends. If it's going to be a mostly passive thing like timezone, it's cool to do. But anything that is primarily a form of cache (like unread_messages) shouldn't be put there. Of course, there's no reason why you couldn't use the themes table if you're careful to store per-user settings anyway (since it's already abused for the custom profile fields)

Let me put it another way. Right now, one of the things I came to regret in WedgeDesk was the way the number of tickets is cached and not stored in the profile of the user. The overhead, despite caching, for that is surprising - and it has other consequences too that I won't get into.[1] So for the next version I'm going to be putting it into the members table.

While at first blush it would make sense to put it into the cache you're proposing or into the themes table, the best solution for me is a new column in the members table. From my perspective, the value is obtained immediately (though as I discussed elsewhere, I don't really like the fact that $user_settings is persisted and contains the *entire* row, regardless of what $user_info has), and I can update it when necessary regardless for users with a single UPDATE, which I can't do in the case of either the cache (because I then have to do one update per user) or the themes table (because I have to establish the value exists to UPDATE it or do an INSERT first)

It's useful for data that's per user but only accessed and updated by that user, or at least, very infrequently by other users.
Quote
I always wondered about that but didn't want to break it.
At least, that's my understanding of it. I can't imagine any other reason.
Quote
Ah, yes, that was it... I remember now how I was frustrated having to cancel my upcoming commit about that...
Quote
We could also simply say -- if your upload process doesn't work, use FTP and don't bother us with that! Just outputting a nice error message explaining what happened is okay to me...
Which means for a fraction of users, they have a cushy ride and everyone else is told to hit the highway. To be honest, I don't feel that's particularly acceptable, it's basically saying "I couldn't be bothered to make this work well for you." in my head.
 1. Suffice to say, SSI has fun with it, and the logic around cache purging when tickets are updated is ridiculous.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #22, on November 3rd, 2011, 03:52 PM »
Quote from Arantor on November 3rd, 2011, 03:17 PM
Quote
Looking at some of your code in updateMemberData...
That was one of mine? Egad, looks ugly. Smells more like something that was there before and that I changed.
Ah, my bad! I just checked in SMF 2, and it's also there, so you have nothing to do with it ;)
I assumed it was your code because it's hook code, and I didn't remember writing it, and SMF has so little hook code that I just didn't even consider they could have written it.

I should have known better :lol:
Quote
I'm a fan of such things, and I'm going to be making use of such when I get to implementing the per-user sanctions (e.g. through warnings and having permissions removed etc.)
Good to know!
Any issues with the field being called 'data', btw...? That's what it is called in my local install. It's not very original and can be confusing, but at least it makes for a nice $user_info['data'] array.
Quote
Let me put it another way. Right now, one of the things I came to regret in WedgeDesk was the way the number of tickets is cached and not stored in the profile of the user. The overhead, despite caching, for that is surprising - and it has other consequences too that I won't get into.[1]
 1. Suffice to say, SSI has fun with it, and the logic around cache purging when tickets are updated is ridiculous.
Well there's definitely some overhead too for the AeMe cache and it's stored in the profile table...
Quote
and I can update it when necessary regardless for users with a single UPDATE,
Yeah, that's the thing with the data field: it's only for when you need to update one member at a time... Which is why the custom function I added for it (updateMyData(), can be renamed later) doesn't take a $members param -- it uses $user_info['id'] directly, at least for now.
Quote
Which means for a fraction of users, they have a cushy ride and everyone else is told to hit the highway. To be honest, I don't feel that's particularly acceptable, it's basically saying "I couldn't be bothered to make this work well for you." in my head.
Do you have any stats, honestly.....? :-/

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #23, on November 3rd, 2011, 04:07 PM »
Quote
Any issues with the field being called 'data', btw...? That's what it is called in my local install. It's not very original and can be confusing, but at least it makes for a nice $user_info['data'] array.
'data' is a little generic, but it works.
Quote
Well there's definitely some overhead too for the AeMe cache and it's stored in the profile table...
There's overhead whatever you do, the only way not to have overhead is to not do anything to the menu in the first place.[1]

But in WD's case, the overhead is actually more trivial; it's not just looking up last-read-id vs last-actual-id or similar, there's a whole bunch of determination of ticket status vs author vs permissions to figure out how many tickets should be shown if the user is staff, and how many if they're a regular user. I spent far too long fighting it and it would save so much effort to just put it in the members table where it can be updated as needed rather than queried and regenerated at best every few minutes (and at worst far more often)

Horses for courses. Do what works best for what you're doing, and in my case I realised too late that I should have put it in the members table.
Quote
Yeah, that's the thing with the data field: it's only for when you need to update one member at a time... Which is why the custom function I added for it (updateMyData(), can be renamed later) doesn't take a $members param -- it uses $user_info['id'] directly, at least for now.
Yup, and that's cool - provided that you are only updating one member at a time. The data field makes that a requirement, not a contextual thing. (You can update multiple rows to SET column = column + 1 in a single query, but you can't update multiple rows based on multiple criteria)
Quote
Do you have any stats, honestly.....?
http://www.simplemachines.org/community/index.php?board=9.0 and http://www.simplemachines.org/community/index.php?board=147.0 searching for 'ftp' and 'chmod'. Those are the users who can't do it out of the box. Pretty much all crap shared hosting (which to be fair, is almost all of it) runs the web server as www-data or nobody or apache, which means it's not writable.

And while there is an argument to be made about using chmod, the idea of making the Plugins folder a+rw doesn't fill me with confidence, especially because it then becomes an attack vector (especially on shared hosts) - it's not like the attachments or avatars folders, where you're expressly locking down content from being executed, the contents of Plugins and its related folders absolutely *must* remain executable. Which means elevating privileges, uploading, then dropping privileges again.
 1. The only winning move is not to play.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #24, on November 3rd, 2011, 11:07 PM »
Quote from Arantor on November 3rd, 2011, 04:07 PM
There's overhead whatever you do, the only way not to have overhead is to not do anything to the menu in the first place.[1]
 1. The only winning move is not to play.
<ahhhhhhhhh, that movie...>
Quote
Yup, and that's cool - provided that you are only updating one member at a time. The data field makes that a requirement, not a contextual thing. (You can update multiple rows to SET column = column + 1 in a single query, but you can't update multiple rows based on multiple criteria)
Sure.

As for the data field... I'm looking into the members table and it's pretty hard finding something that isn't used in a regular query.
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.

passwd_flood may be a candidate. I'd like to invite you to look into it... Try a global search for it. In some situations, it's retrieved from the database, but never used later. Look inside Reminder.php for instance. My guess is that it was planned to be used in validatePassword() (again, I'd suggest looking through all occurrences of validatepassword.) This may be an actual SMF bug...

secret_answer/question is a strong candidate too. It's used in a regular query, but it's always LIMIT 1 so it's worth doing an unserialize on it every time. Same problem as pm_prefs though, for saving.

Anything to justify having this new field, really... :lol:
Quote
Pretty much all crap shared hosting (which to be fair, is almost all of it) runs the web server as www-data or nobody or apache, which means it's not writable.
Even if it's running as www-data etc, I believe there are some ways to configure the server to make it work...

I use MANY hosts in the past. Only one had issues with modifying the filesystem through PHP, IIRC.

PS: feel free to fix any bugs you find in the thought system, now that it's committed. 'Modify' doesn't work anymore, I'll look into it tomorrow.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #25, on November 3rd, 2011, 11:11 PM »
Quote
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.
Yeah, that one's pretty mindblowing to understand.
Quote
passwd_flood may be a candidate. I'd like to invite you to look into it... Try a global search for it. In some situations, it's retrieved from the database, but never used later. Look inside Reminder.php for instance. My guess is that it was planned to be used in validatePassword() (again, I'd suggest looking through all occurrences of validatepassword.) This may be an actual SMF bug...
I will look at that, I never knew what the field did anyway.
Quote
secret_answer/question is a strong candidate too. It's used in a regular query, but it's always LIMIT 1 so it's worth doing an unserialize on it every time. Same problem as pm_prefs though, for saving.
It's created in one place, it's only used in one place, I'm not seeing what the problem is?
Quote
Even if it's running as www-data etc, I believe there are some ways to configure the server to make it work...
Yes... making files writable by default. Making an area of your space, that's going to be executable by live request, open to be modified by anyone else on the server.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #26, on November 3rd, 2011, 11:19 PM »
You probably missed the PS I added after my post :)
Quote from Arantor on November 3rd, 2011, 11:11 PM
Quote
Some things like pm_prefs could be transferred, but they're set through a profile array that's pretty complicated and implies saving the variable to the members table itself... I'm not thrilled at the idea of tweaking that.
Yeah, that one's pretty mindblowing to understand.
The list? It's probably a good example of how a good idea (gathering similar data saving into a single array for easier maintenance) gets blown out of proportion and becomes totally unmaintainable because of silly validation functions etc. It's documentation in the comments is even more confusing. As if it was all written to be maintained only by its original writer...
Quote
I will look at that, I never knew what the field did anyway.
Good opportunity ;)
Quote
It's created in one place, it's only used in one place, I'm not seeing what the problem is?
No problem, except for the saving of it.
Quote
Yes... making files writable by default. Making an area of your space, that's going to be executable by live request, open to be modified by anyone else on the server.
Hmm alright...

Okay, going to bed early tonight. I'm tired... I think I've never spent so much time changelogging a feature. I must have re-re-re-reviewed the entire file list about 10 times in a few days...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #27, on November 3rd, 2011, 11:36 PM »
Quote
You probably missed the PS I added after my post
I did, yes.
Quote
The list? It's probably a good example of how a good idea (gathering similar data saving into a single array for easier maintenance) gets blown out of proportion and becomes totally unmaintainable because of silly validation functions etc. It's documentation in the comments is even more confusing. As if it was all written to be maintained only by its original writer...
It's a true SMF variable then.
Quote
Hmm alright...
It's not for nothing that vB and xF store their templates in the database...
Quote
Okay, going to bed early tonight. I'm tired... I think I've never spent so much time changelogging a feature. I must have re-re-re-reviewed the entire file list about 10 times in a few days...
Sleep well :)

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Thought system
« Reply #28, on November 6th, 2011, 12:11 PM »
Just wanted to say that it took one more week than expected, but I'm hoping to have the thought system implemented by tonight... Or tomorrow if I'm too tired (only got 5 hours sleep... -_-)

So it's basically what we have on Wedge.org, except that:
- you can edit your current thought or create a new one (on wedge.org, if you want to edit your thought you need to edit it word by word or the site will mistake it for a new thought),
- it has privacy settings (no changes compared to the current SVN though...),
- you can reply to any thought (need to consider whether they should inherit their parent's privacy setting... there's no quote so it's not a BIG deal for now), but you can also delete any of your thoughts (or any thought if you're a global moderator), and edit your own replies.

The code is relatively ugly for now... I had to make concessions and I hope I will be able to get rid of them later.
Also, you can no longer simply click a thought to answer it. You have to click the Reply button next to it -- not because it's impossible, but because it makes more sense to click a thought to *edit* it, and because I don't want to make the mistake of editing a third party's thought instead of replying it (as the admin), I selfishly made it impossible to click directly...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Thought system
« Reply #29, on November 6th, 2011, 02:12 PM »
There's a side benefit of that for touch users, if someone posts a link as a thought, it means it's accessible directly as opposed to tapping once to activate edit and subsequently disable the event handler, so that the link becomes accessible directly.

I think the choice of forcing a discrete button press is not a bad one, you still have the notion of a button press anyway but it's much clearer as to what you're doing :)