Mad idea but it might just work

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Mad idea but it might just work
« on March 7th, 2012, 05:54 PM »
OK, so I've been trying to figure out how to cope with this. One of the biggest problems with SMF's environment is that it makes your files writable by everything, essentially. You have to make your site vulnerable on shared hosting in order to upload and install mods - even pure hooks ones.

In our case, there's no file edits at all. But there's still the upload problem: you have to write something to the relevant folder. Which means that folder has to be writable by the webserver.

So, I tried a different tact, what if through some process, we no longer required write access to that folder, but that we could delegate it off to something which did? Or, as I call it, you upload the plugin file, it goes into the system temp folder (where all uploaded files go), where it's unpacked.

Here's the trick: instead of uploading it into a physical folder or something like SMF does, we unpack it serially (going through the archive file by file), push the file to temp, then we send it via FTP from our script to the server. It's seemingly stupid but I see no reason why it won't work, other than the mechanics are a PITA.

Doing that means you don't have to make the folder writable or indeed do anything to it, essentially it's being uploaded to by you (and thus OWNED by you) just as if you uploaded it yourself. No changing permissions, no changing them back.


The caveat is that I would remove direct local filesystem support. This would make for a slight inconvenience on test forums/localhost where FTP isn't configured, because it would mean there would be no facility for uploading as SMF currently does. But the price paid in convenience is security: if you don't ever have direct filesystem support, there's no need to screw around with making things 777 (and hopefully that plague of bad advice will not follow us here), and it's not like you can't just unpack it and upload it yourself.

I have the uncomfortable feeling it would pretty much demand .zip support because the contents of /tmp are intentionally unstable and I'm not sure how comfortable I am with unpacking a .tar.gz in /tmp and expecting it all to be there after (as opposed to .zip which can be handled a file at a time)


So, thoughts? Concerns? Questions? Anything that didn't make sense and people would prefer I explained it in actual English?
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Mad idea but it might just work
« Reply #1, on March 7th, 2012, 06:34 PM »
Concern, yes: this does mean Wedge has to store FTP credentials, doesn't it...? To me, it's more... Interesting as data, than access to the folders themselves.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #2, on March 7th, 2012, 06:43 PM »
No more than SMF does: store the username, port and maybe the partial-path[1] in the database, and request the password once per session (and store that obfuscated in session)
 1. You know, where it presents /home/user/ as /

Farjo

  • "a valuable asset to the community"
  • Posts: 492
Re: Mad idea but it might just work
« Reply #3, on March 7th, 2012, 06:49 PM »
Well I understand all the works but not necessarily them all collectively :hmm: Will it mean we won't be able to load mods on a test version or will it just be a bit of a faff on the part of the programming?

As for the rest I cannot really comment.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Mad idea but it might just work
« Reply #4, on March 7th, 2012, 06:50 PM »
@Pete> Yes, but SMF doesn't *need* the FTP data if you're going to install a mod. It only asks for it if your server has permission issues...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #5, on March 7th, 2012, 06:57 PM »
OK, here's the deal, if the server has it writable already, there is a security issue. No ifs, buts or maybes, you have a security risk. On a local test site, it doesn't matter, because only you have access to the machine.

But on a public server with multiple users (or multiple, but separated sites), that's a risk. Why do so many people complain on sm.org about being hacked? Because their forum is mostly left writable.

By forcing it to go through an FTP tunnel, you don't ever change permissions, you don't ever make the files owned by the webserver (which is almost as insecure) and you don't have to contend with chmod. At all.

The downside is that forthe cases where you don't care about the risk, the convenience can't be given, because users who don't know what they're doing will blindly reconfigure it, on receipt of bad advice, and proceed to repeat that bad advice. So many people at sm.org simply do not understand the consequences of what they're suggesting.

For test sites, sure, you can't magic upload, but you can just unpack the zip directly into Plugins/. There's nothing more the uploader needs to do, other than literally unpack the files, no install process, no testing (that all happens prior to enable)

Farjo

  • "a valuable asset to the community"
  • Posts: 492
Re: Mad idea but it might just work
« Reply #6, on March 7th, 2012, 07:24 PM »
I see your point about security, and indeed about passing on bad advice.

As regards techie-ness I'm probably the median average admin. I understand chmod, but not what the permissions should be and I'm not happy about that - now that our club is becoming more popular I worry that we will attract the wrong sort of attention. So any increase in security is welcome news.

When I want to look at a mod I use our test copy - if mysite.com is our live address then it's mysite.com/test. I load the mod then I basically dick around with it to see if it does what I expected and to see if I can break it. Presumably I'd still be able to do this?

Unzipping a file and ftp-ing stuff to the ./plugin/ directory is not a problem, however I'd then like it if it was the same process on live, otherwise (in my mind) I'm introducing a component of the process that I have not tested. But I doubt anyone else would see it this way  :)

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #7, on March 7th, 2012, 07:26 PM »
In your case, you'd use exactly the same process. The only time for using it yourself is if you run a local test server that doesn't have FTP or SFTP on it.

And while I understand your concern about it being two different processes, it really isn't, the actual install doesn't happen till you hit the enable button, so it really is automating the upload process that we're talking about here.

Farjo

  • "a valuable asset to the community"
  • Posts: 492

Powerbob

  • Posts: 151

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #10, on March 8th, 2012, 03:00 PM »
OK, so a little progress update from me. I'm curious how this is going to work going forward, because in the not too distant future, it can't work.

To explain briefly, when you create an FTP connection of the kind needed here[1], several things happen. Firstly, it isn't one connection that's made, it's two. You have a control connection, which is the one that you normally see in an FTP client, it's the one where the text instructions are sent.

Then you have a data connection, which operates on another port, and sometimes even another IP address, where the data goes to/from. There are several neat things you can do with this.[2]

Here's the interesting quirk. When you create that secondary connection, you get a string back from the remote server, in the form of a,b,c,d,x,y (6 numbers, separated by commas), a-d are the four parts of the IP address and x-y are the port number, and you have to multiply x by 256 and add y to get the number back.

Anyone see the problem yet?

There is absolutely zero IPv6 support in the FTP specification. At least, in the classic RFC 959 from 1985 that outlines FTP.

Now, RFC 2428 from 1998 does explain how to support IPv6, but I'm not clear how widespread the use of EPSV vs PASV is (this is the instruction required to start a passive connection, which is very specifically what we need to use), so I'm honestly not 100% clear on how feasible it is to support these.

On top of this, it gets worse. There isn't a single consistent method for handling FTP connections. There's an FTP extension to PHP, there's the partial FTP connector in SMF/Wedge (which does not include a send-file method), plus a variety of other connectors too, all of which with their own foibles to contend with.

I'm not giving up just yet, but I've spent a lot of time reading up on these specifications, learning far more minutiae than I *ever* wanted to know about FTP, and this is before I even tackle SFTP. I just need to step back and clear my mind before I try implementing anything on this.
 1. In other words, one where you're sending or receiving data, and not just changing directories, etc.
 2. Namely you can have three machines, A, B and C. A is the master controller, and deals with having control connections to B and C, then A can tell B to send C a file, and tell C to receive it, while A itself doesn't ever have to do anything other than just delegate.

billy2

  • Trying to earn brownie points for a lads trip to the Red Sea. Minus 1 already - just for asking!!
  • Posts: 350
Re: Mad idea but it might just work
« Reply #11, on March 8th, 2012, 03:28 PM »
Never knew so much went on when I connect CuteFTP  :yahoo:

STATUS:>   Connect: Thursday 14:22:08 03-08-2012
STATUS:>   Connecting to ftp.mysite.co.uk
STATUS:>   Connecting to ftp.Mysite.co.uk (ip = x.x.x.x)
STATUS:>   Socket connected. Waiting for welcome message...
   220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
   220-You are user number 1 of 50 allowed.
   220-Local time is now 14:22. Server port: X.
   220-This is a private system - No anonymous login
   220-IPv6 connections are also welcome on this server.    <<<<<<==== never read this before
   220 You will be disconnected after 15 minutes of inactivity.
STATUS:>   Connected. Authenticating...
COMMAND:>   USER X
   331 User X OK. Password required
COMMAND:>   PASS Y
   230 OK. Current restricted directory is
STATUS:>   Login successful
COMMAND:>   TYPE I
   200 TYPE is now 8-bit binary
STATUS:>   This site can resume broken downloads
COMMAND:>   PWD
   257 "k" is your current location
COMMAND:>   TYPE A
   200 TYPE is now ASCII
COMMAND:>   REST 0
   350 Restarting at 0
STATUS:>   Retrieving directory listing...
COMMAND:>   PASV
   227 Entering Passive Mode (c,c,c,c,c,c)
COMMAND:>   LIST
STATUS:>   Connecting data socket...
   150 Accepted data connection
   226-Options: -a -l
   226 49 matches total
STATUS:>   Received 3662 bytes Ok.
STATUS:>   Time: 0:00:01, Efficiency: 3.58 KBytes/s (3662 bytes/s)
STATUS:>   Done.
<br /><br />cough, cough.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #12, on March 8th, 2012, 03:33 PM »
Well, the 220 lines are just informational; IPv6 may be welcome but it requires more work to set it up, and I can tell immediately from that you're still using IPv4 yourself (which is what the PASV and the status 227 indicates)

But yeah, I can now tell you what all that crap means.

billy2

  • Trying to earn brownie points for a lads trip to the Red Sea. Minus 1 already - just for asking!!
  • Posts: 350

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Mad idea but it might just work
« Reply #14, on March 8th, 2012, 03:42 PM »
I'm not pissed that I had to find out. I'm pissed that it feels like such a clusterfuck.

It makes the HTTP specification look even more sane than I thought it was, which is an achievement.