Wedge

Public area => Bug reports => The Pub => Archived fixes => Topic started by: Pandos on January 28th, 2014, 01:19 AM

Title: Issues with CDN's
Post by: Pandos on January 28th, 2014, 01:19 AM
If you are switching to an CDN (e.g. Google, Cloudflare) there are some issues to resolve:
@MultiformeIngegno: please keep track if there are more issues with CF. I'll check Google PSS.
Title: Re: Issues with CDN's
Post by: MultiformeIngegno on January 28th, 2014, 02:26 AM
I can confirm the login (session verification failed) issues (had with SMF as well). My IP anyway is not the same of the CDN, as I set this(https://support.cloudflare.com/hc/en-us/articles/200170706-Does-CloudFlare-have-an-IP-module-for-Nginx-) up on nginx (and in fact in logs real IP are present, not CDN ones):

Code: [Select]
   set_real_ip_from   199.27.128.0/21;
   set_real_ip_from   173.245.48.0/20;
   set_real_ip_from   103.21.244.0/22;
   set_real_ip_from   103.22.200.0/22;
   set_real_ip_from   103.31.4.0/22;
   set_real_ip_from   141.101.64.0/18;
   set_real_ip_from   108.162.192.0/18;
   set_real_ip_from   190.93.240.0/20;
   set_real_ip_from   188.114.96.0/20; 
   set_real_ip_from   197.234.240.0/22;
   set_real_ip_from   198.41.128.0/17;
   set_real_ip_from   162.158.0.0/15;
   set_real_ip_from   2400:cb00::/32;
   set_real_ip_from   2606:4700::/32;
   set_real_ip_from   2803:f800::/32;
   set_real_ip_from   2405:b500::/32;
   set_real_ip_from   2405:8100::/32;
   real_ip_header     CF-Connecting-IP;

WordPress' sessions work just fine.. maybe yeah Wedge/SMF should have some edits regarding HTTP_X_FORWARDED_FOR ?

I did not test the rest yet.
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 10:11 AM
I think it's not done with HTTP_X_FORWARDED_FOR in Wedge.
As far as I remember on SMF there are changes to QueryString.php to pick up the real IP address
member_ip1 is host ip.  member_ip2 is members real ip, etc....

One thing I already discussed with Nao is that previews of attachments should have the right extension to get served by a CDN.

Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 10:12 AM
Quote from MultiformeIngegno on January 28th, 2014, 02:26 AM
I can confirm the login (session verification failed) issues (had with SMF as well). My IP anyway is not the same of the CDN, as I set this(https://support.cloudflare.com/hc/en-us/articles/200170706-Does-CloudFlare-have-an-IP-module-for-Nginx-) up on nginx (and in fact in logs real IP are present, not CDN ones):

Code: [Select]
   set_real_ip_from   199.27.128.0/21;
   set_real_ip_from   173.245.48.0/20;
   set_real_ip_from   103.21.244.0/22;
   set_real_ip_from   103.22.200.0/22;
   set_real_ip_from   103.31.4.0/22;
   set_real_ip_from   141.101.64.0/18;
   set_real_ip_from   108.162.192.0/18;
   set_real_ip_from   190.93.240.0/20;
   set_real_ip_from   188.114.96.0/20; 
   set_real_ip_from   197.234.240.0/22;
   set_real_ip_from   198.41.128.0/17;
   set_real_ip_from   162.158.0.0/15;
   set_real_ip_from   2400:cb00::/32;
   set_real_ip_from   2606:4700::/32;
   set_real_ip_from   2803:f800::/32;
   set_real_ip_from   2405:b500::/32;
   set_real_ip_from   2405:8100::/32;
   real_ip_header     CF-Connecting-IP;

WordPress' sessions work just fine.. maybe yeah Wedge/SMF should have some edits regarding HTTP_X_FORWARDED_FOR ?

I did not test the rest yet.
Simple question:
Did it solve your session errors with wedge?
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 10:43 AM
Just want to be sure of something...
Did you guys both enable proxy settings in Wedge..?

Admin > Server > Server (1st choice) > Proxy Settings.
I'm just asking.
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 10:50 AM
I haven't tested Wedge on PSS so far.
Proxy settings looks promising for the session thingy.

Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 11:16 AM
Well, it's the goal... IIRC, Pete wrote that for Bad Behavior handling (BB actually searches for CF headers), but it can be applied to any reverse proxy in general, and from what I can gather, CloudFlare and PSS would belong to that category.

Oh, I just looked through the source code, and noticed that CloudFlare is even mentioned in the comments!
So, basically:
- Take grepWin or any program that does folder-wide searches,
- Search for CloudFlare,
- And there you have it, your solution. ;)

I'm just surprised that Lorenzo wouldn't know about the feature, considering he's been using CF for quite some time now. I'm sure it was discussed before.
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 11:18 AM
OK, the biggest culprit is gone :)
But there are still remaining some issues.
What's your opinion to the search issue?
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 12:02 PM
Hmm...
There's a 2083-char limit for IE requests, but I don't know anything about reverse proxies having problems with char limits as well.

The thing is, the original search is done through a post, so I don't see how it could be a problem. The only way I can get $context['params'] to show up is through a 'Next Page' link, for instance. Is that where you start having problems..?
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 12:12 PM
Google limits the maximum length of URL's to 2K (2048).
So there's a problem if you do a search and want to display the "Next Page".
https://code.google.com/p/googleappengine/issues/detail?id=7053
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 12:15 PM
But that's only *there*, right..? Search isn't entirely broken to begin with?
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 12:17 PM
Entirely broken is not the right wording.
It only affects search results after the initial page 1.
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 12:18 PM
Worst I could get in the Search Results linktree was about 265 bytes. Add another dozen bytes for ';start=30', and... Well, it's a far cry from the 2K bytes I was fearing I'd get.

Seriously, how can you get super-long query strings then..?!
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 12:27 PM

Actual search from SMF driven forum: about 3254 bytes
Code: [Select]
/index.php?action=search2;params=YWR2YW5jZWR8J3wwfCJ8YnJkfCd8NjgxLDMzLDMzNywzMzgsNTAsMjQ1LDYzOCw3OSwyMTIsMjEzLDIxNCwyMTUsMjE2LDIxNywyMTgsMjE5LDIyMCwyMjEsMjIyLDIyMywyMjQsMjI1LDIyNiwyMjcsMjI4LDIyOSwyMzAsMjMxLDIzMiwyMzMsMjM0LDIzNSwyMzYsMjM3LDYxLDEwNywyOTQsMzExLDMxNywzMTgsMzIwLDMxOSwzMjEsNjQ1LDI1LDU2MCw1NjEsNTc5LDU2Miw1NzQsNTYzLDU3NSw1NzYsNTc4LDU3Nyw1NjQsNTY1LDU2Niw1NjcsNTY4LDU3MCw1MjgsNTcxLDU3Miw1NzMsNjAwLDY0NywxMTAsMjAsNTQ5LDYyNyw2MjgsNjA3LDYwOCw2MTEsNjEyLDYxNCw2MTMsNjE1LDYxNiw2NDEsMjkzLDYwOSw2MTAsNTkyLDI0MiwyNDcsMjQ4LDU5Myw2MjAsNTk0LDYxOCw2MDYsNjE3LDYzOSw2NDAsNjcxLDY3Myw2MTksMjc4LDU5NSw1OTYsMzIsNTk5LDUsNTk4LDY4NywxMywxNDIsMjY5LDI3MCwyNzEsMjcyLDI3MywyNzQsMTQzLDMwMiw1MjMsMTQ0LDE0NSwzMzEsMjQ5LDg1LDU4OSw2MjMsNTkwLDYyNCw1OTEsNjI1LDUyNyw1MzAsMTEsNTg1LDU4Niw1ODcsNTg4LDIwNCw1OTcsNjAxLDM3LDEwNiw0LDE2NCwzMzUsMTYzLDQxLDczLDIwOSwxNjUsMTY2LDE2NywxMzEsNjM2LDYzNyw2NjUsMTI5LDI0MCwyNTQsNjAyLDYwMyw2MDQsNjA1LDMwMSw1NDgsNTM4LDMyMiw1MzksNTQwLDY2MSw2NjIsNjc5LDU0MSw1NDIsNTQzLDU0NCw1NDUsNTQ2LDEwLDEyNiwxMzYsNjQ4LDEyNyw2OTEsODksMTAzLDY3LDUzMSwyMSw1NTIsNTU1LDMxNSw1NTYsMTY4LDU1Myw1NTcsNTUwLDU1OCw1NTEsNTU5LDY5LDU1NCw2NDQsNjQ2LDY4LDIyLDczMyw3MzUsNzM3LDM0Myw2NTgsNjU5LDI5MiwzNDIsMzMyLDIzLDcxMyw3MTUsNzE3LDcxOSw3MjEsNzIzLDcyNSwxOSw0Nyw2NiwxNzIsMzU5LDQ2NSw3MDEsMTY5LDE3MCwzNTYsMTcxLDM1NSwyMDYsMzY5LDE3NiwyMDIsMzU4LDQ2Niw0NjcsNDY4LDE3NCw1MjYsMTc1LDMwMywzNjEsNTMyLDE3MywzNjAsMTk2LDY1Miw2NTMsNjU0LDM2OCwxNzcsNTM3LDE5Nyw1MzYsMTk4LDIwNyw2MjYsNzA5LDcwMyw3MDcsNzI3LDY5NywzLDMyMywzMjQsMzI1LDMyNiwzMjcsMzI4LDM0MCwzNDEsMzYzLDM2MiwzNjQsNDEyLDk5LDYwLDgzLDE4Miw0OCw1OSw1NCw1Niw1OCwxMzQsMTExLDM0NiwzNDcsMzQ5LDM1MCwzNTEsMzUyLDE4MywzOCw2OTksOTgsNTAxLDUwMiw1MDcsNTA4LDUwOSw1MTAsNTExLDUxMiw1MTMsNTE0LDUxNSw1MTYsNTE3LDUxOCw1MTksNTIwLDUyMSw1MjIsNTAzLDUwNCw1MDUsNTA2LDk3LDQ4NSw0ODYsNDg3LDQ4OCw0ODksNDkwLDQ5MSw0OTIsNDkzLDQ5NCw0OTUsNDk2LDQ5Nyw0OTgsNDk5LDUwMCw5NSwxODYsMjQ2LDI5OCw5NiwyOTksMzY2LDM2NywzNTMsMTE4LDExOSwxOTMsMTk0LDMxNCwyNTUsMjQzLDEzNSw4NCw4MSwzMzksNjYzLDEyMywxMzAsMjUwLDExMiw2ODUsMTA5LDEyMSwxMzgsMTM5LDY3NSwxLDM3MCwyMzgsMzA4LDQ1LDQ2MywyMTEsMjY0LDI2NSw1MjUsMjY2LDUyNCw0NjQsMjA1LDI4LDE0MCwyNDQsOTEsNjc3LDM5LDMwNCw4Niw1ODQsMTE1LDEyMiwyMDgsNjQ5LDY3MCw5MywxNjIsNDcxLDQ3MiwzNjUsNDc5LDQ4MCw0ODEsNDgyLDQ4NCwxNTYsNDY5LDQ3MCw0NzMsNDc1LDQ3Niw0NzcsNDc4LDY4OSw2NDMsNjUwLDgsMTAxLDI5Nyw2MzIsMTMzLDYzNSwxNDcsMTQ4LDE0Niw1ODMsNTgwLDYyOSw1ODIsNjMxLDYzMCw1ODEsMjEwLDYzMyw2MzQsMzA5LDMxMCw5LDMzMCw1MzMsNTM0LDUzNSw2NjYsNjY3LDY2MCwzMTYsMTMyLDMyOSwzMzYsNTQ3LDcwNSw3MjksNzMxLDY2OCw2ODMsMzcxLDM3Miw2NjksMzczLDM3NCwzNzUsMzc2LDM3NywzNzgsMzc5LDM4MCwzODEsMzgyLDM4MywzODQsMzg1LDM4NiwzODcsMzg4LDM4OSwzOTQsMzk1LDI5NSw2OTMsMzk4LDM5OSw0MDAsNDAxLDQwMiw0MDMsNDA0LDYyMSw0MDUsNDA2LDQwNyw0NDEsNDQyLDQ0Myw0NDQsNDQ1LDQwOCw0NDYsNDQ3LDQ0OCw0NDksNDA5LDQ1MCw0NTEsNDUyLDQ1Myw0NTQsNDEwLDQ1NSw0MTEsMTA4LDE4MCwxNzgsMzA1LDMwNywzMDYsMTAyLDM1NCw2OTUsNTEsNDE1LDQxNiw0MTcsNDE4LDQxOSw0MjAsNDIxLDQyNyw0MzAsNDU4LDQ1Nyw0MzEsNDMyLDQzNyw0MzQsNDM1LDQzNiw0MzN8InxzaG93X2NvbXBsZXRlfCd8fCJ8c3ViamVjdF9vbmx5fCd8fCJ8c29ydHwnfHJlbGV2YW5jZXwifHNvcnRfZGlyfCd8ZGVzY3wifHNlYXJjaHwnfGNyZWFkb28;start=20
Actual search from wedge is about 222 bytes:
Code: [Select]
http://wedge.org/do/search2/?params=eJwtjUsOwjAMBe_Chg0L3JTfaaI2sRQgNChtQUg5PGOpi5Ht0bM91tj2TeR2kO4IHTjo4QRnuMAVyDi8wzu8s5xQDTKOXTHPHbGeHcELeenbrs2pfH0or3fWRflqah0fGhZfpvzbTKmLj_fKFHUOm2GqmvUzTEFN6VBDQqZSnn-VTDmW;start=30


But:
It depends on how many boards do you have to search. Each board is given as a parameter in this URL. I have about 640 boards...
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 12:43 PM
I see...
Well, I have a potential fix for that, but I'm not sure it'll work. Stay tuned.
Title: Re: Issues with CDN's
Post by: MultiformeIngegno on January 28th, 2014, 12:46 PM
:o I really didn't remember the proxy setting. I'll try it soon
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 01:00 PM
:cool:
Posted: January 28th, 2014, 12:56 PM

Pandos, I've sent you a PM about this!
Title: Re: Issues with CDN's
Post by: MultiformeIngegno on January 28th, 2014, 01:18 PM
Now with CF on sessions work fine even without proxy setting on. I don't know what to say.. <_<
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 01:21 PM
Lol.

Well, I think that's just a random thing anyway. (i.e., hard to reproduce.)

Did you test the rest of the bugs mentioned by Pandos in his first post?
I'm currently working with him in private to see if my alternative handling of the query string can fix the search problem.
Title: Re: Issues with CDN's
Post by: MultiformeIngegno on January 28th, 2014, 06:29 PM
Will test the other issues tonight! Any news on the mysql deprecated function of the importer?
Downside of being on latest php ... :P
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 06:59 PM
Well, I got started on it earlier today, and then got pissed at how annoying it is to convert... You know, some conversions are no-brainers (just add an 'i'), others require switching params... Meh!!
I'll probably finish that tomorrow.
Title: Re: Issues with CDN's
Post by: Nao on January 28th, 2014, 07:01 PM
Quote from Pandos on January 28th, 2014, 01:19 AM
  • Attachments not showing up
    It looks like those resources are being served out of the reference domain
     with a "Content-Encoding: none" header. This causes some problems with  decoding the response (confirmed by Google).
    But this can be easily fixed.
How..?
Title: Re: Issues with CDN's
Post by: Pandos on January 28th, 2014, 07:15 PM
Quote from Nao on January 28th, 2014, 07:01 PM
Quote from Pandos on January 28th, 2014, 01:19 AM
  • Attachments not showing up
    It looks like those resources are being served out of the reference domain
     with a "Content-Encoding: none" header. This causes some problems with  decoding the response (confirmed by Google).
    But this can be easily fixed.
How..?
As far as I can see this affects  AeMe too. Just remove Content-Encoding: none and everything is OK :)
Or more elegant: Content-Encoding: image/jpg, png, etc...

In SMF it was in QueryString.php.
Title: Re: Issues with CDN's
Post by: Nao on January 29th, 2014, 01:03 PM
But it's there for a reason, innit..?
Title: Re: Issues with CDN's
Post by: Pandos on January 29th, 2014, 01:15 PM
IMHO it's for compatibility reasons?
You can upload different types of files. To ensure it's not an e.g. executable file, it's obfuscated.
But there's really no need for it?
What should speak against leaving the attached filenames as they are?
They all get obfuscated with a file hash.


This situation makes it really impossible to easily get content to social media networks.
Also serving with the correct content-type is the best and secure thing.
Title: Re: Issues with CDN's
Post by: Pandos on January 29th, 2014, 01:22 PM
By the way. Attachments are already stored with the correct mime type in db. Correct me when I'm wrong, but I see no culprit in it.
Title: Re: Issues with CDN's
Post by: Nao on January 30th, 2014, 12:46 PM
I'm a little slow here. I don't get what you're trying to make me remove, and why you're assuming that it's not needed. :^^;: