JavaScript caching

Nao

  • Dadman with a boy
  • Posts: 16,082
JavaScript caching
« on May 10th, 2011, 06:43 PM »
Feature: JavaScript caching
Developer: Nao
Target: everyone
Status: 100% (believed to be complete.)
Comment:

Right after minifying your CSS and JS files, Wedge will cache them and gzip them. The good point is that it's smart enough to cache multiple files together, relieving the server a bit.

I'll use my script.js example again. At the time of writing, the regular script.js/theme.js duo is about 50KB in SMF2 (split over two files), less than 30KB in Wedge (with many extra functions in them), and the actual stuff you download is a single 6KB file, after minification and gzipping. Meanwhile in the wasteland, SMF2 still sends you 50KB of data. Because everyone knows all end-users absolutely need to have comments in the JavaScript code they execute.

Re: JavaScript caching
« Reply #16, on July 3rd, 2011, 05:44 AM »
Dragoon, although I wish you would have asked me before quoting that, it is my post and I don't mind (for that one).

Arantor, yeah, that one.

Re: JavaScript caching
« Reply #17, on July 4th, 2011, 05:11 AM »
Quote from Nao/Gilles on July 1st, 2011, 02:17 PM
I originally minified everything together, but performance was not any better. Probably a bit worse IIRC. Plus, it had other issues linked to it. Once I started doing files separately, it went better.
Hmm, this is not the same as my experience, odd.  I just combine with newlines between the files, which generally solves all the problems I can think of (some people don't add newlines, which means Windows-style files with a comment at the end or a missing semicolon can have issues.)
Quote from Nao/Gilles on July 1st, 2011, 02:17 PM
Also, performance is not very important at the minification stage.
Sure, unless you're debugging/developing.
Quote from Nao/Gilles on July 1st, 2011, 02:17 PM
Commenting JS files should be mandatory. If only to help future developers get into the groove :)
Well, with minification, yes.  Although, I see the JS has changed markedly since I last touched it.  I've mentioned before my dislike of hungarian type notation.
Quote from Arantor on July 3rd, 2011, 01:57 AM
Oh, I remember that discussion now. It was about what JavaScriptEscape did, and why it was necessary to call it for URLs - because there were all kinds of bad URLs getting into Google (namely the one indicated with %msg_id% in it)
Ah, I've seen this too.  I think the best way (and the way I use) is to just use a common base url variable.  I think it's better to json encode values (and not have to deal with the quotes and escaping and such) than this JavaScriptEscape business.  Note that json_encode escapes / as \/ to avoid the </script> issue, which is cleaner and more obvious than '<' + '/'.

Also AFAIK Google appears to often aggressively detect URLs, so even escaping the href probably does no good (at least in my usages it didn't.)

Last of all, \t can be cleanly represented in js and \n can be escaped as in:

var x = '\
   <div>\
     Hi\
   </div>';

So if JavaScriptEscape is kept, it's probably better to just escape the newline rather than making it \n, if concerned about readability.  Probably gzips better.

Of possible interest to this topic, I've been briefly experimenting with a plugin to TOX-G that adds minification to inline js/css (as long as it doesn't have any ifs/etc. in it) and minifies some parts of the HTML (all of this done in the cached template, so runtime speed is not affected.)  Initial impressions seem like it's mostly a waste of time (at least in my inline script tags, which are not common), but offers a small benefit.  I haven't been messing with it much since.

-[Unknown]

Re: JavaScript caching
« Reply #18, on July 4th, 2011, 04:21 PM »
I think minification is more about saving space in the cache. Especially for mobile devices.

Re: JavaScript caching
« Reply #19, on July 4th, 2011, 10:15 PM »
Quote from groundup on July 4th, 2011, 04:21 PM
I think minification is more about saving space in the cache. Especially for mobile devices.
In my experience, reducing HTTP requests has a significant effect even on desktops.  And since it normally seems to cut 20%, and mean that I'm not afraid of leaving potentially concerning comments all over my js code, it seems good for inline script too.

One of the "good" reasons for using inline script is when it's only used on the one page, it's an uncommon page, and you want to reduce the HTTPs even further, especially if the script is small.  But lowering the network consumption is a good thing, because if I can reduce the whole page by 20% (which I can't, more like 8% it seems from a standard HTML page even with some embedded script), without any additional CPU cost, that can potentially be a good savings both for user and the network operator who has to be $5/megabit or whatever.

I've become more concerned about these issues as more and more US companies have been limiting their users' bandwidth, which I hate.

-[Unknown]

Re: JavaScript caching
« Reply #20, on July 5th, 2011, 07:01 PM »
Quote from [Unknown
link=msg=263516 date=1309749092]
Hmm, this is not the same as my experience, odd.  I just combine with newlines between the files, which generally solves all the problems I can think of (some people don't add newlines, which means Windows-style files with a comment at the end or a missing semicolon can have issues.)
I think I accounted for that...
Quote
Sure, unless you're debugging/developing.
True dat. But OTOH, if you're debugging/developing, generally you won't be enabling the minifier at all.
If you're specifically testing the minification process, then you probably won't need very high performance -- if it works, no need to re-parse.
Quote
Well, with minification, yes.  Although, I see the JS has changed markedly since I last touched it.  I've mentioned before my dislike of hungarian type notation.
I know, but it's there, so... I just use it. It's a matter of being consistent with the existing codebase. There would be too much to change otherwise.
Also, sometimes it can be helpful to know what the *intended* purpose of a variable is. I know it has helped me... Don't forget that our code needs to be readable by future devs!
Quote
Ah, I've seen this too.  I think the best way (and the way I use) is to just use a common base url variable.
You mean like <base href>...?
Quote
I think it's better to json encode values (and not have to deal with the quotes and escaping and such) than this JavaScriptEscape business.
But it would mean having to decode after that, wouldn't it...?
I don't even remember seeing a ready-made function for that in ECMAScript...
Quote
Note that json_encode escapes / as \/ to avoid the </script> issue, which is cleaner and more obvious than '<' + '/'.
Indeed. But maybe Google automatically decodes these, as opposed to the + business..?
Quote
Also AFAIK Google appears to often aggressively detect URLs, so even escaping the href probably does no good (at least in my usages it didn't.)
Yeah, I don't think it does.
I can find a middle point by turning href into hr\ef... (Seems to work, at least.)
Quote
Last of all, \t can be cleanly represented in js
It seems to be the case for me... In which case, why did JSE ever begin to escape them...?
Quote
and \n can be escaped as in:

var x = '\
   <div>\
     Hi\
   </div>';
Yes, but it doesn't save a byte, and because of \t, the layout doesn't look good. Although at the time I didn't notice \t could be printed out directly as a tab in the code...
Quote
So if JavaScriptEscape is kept, it's probably better to just escape the newline rather than making it \n, if concerned about readability.  Probably gzips better.
Gzip doesn't really matter here, you'd be surprised with the strange results I've been getting. (There's a recent discussion somewhere about the effort I went into converting some variable names to be one char instead of long stuff, and in the end the resulting gzipped file was actually longer by a few bytes... I just couldn't believe it.)
Ah, would be swell if Apache added support for 7z or even RAR... :lol:
Quote
Of possible interest to this topic, I've been briefly experimenting with a plugin to TOX-G that adds minification to inline js/css (as long as it doesn't have any ifs/etc. in it) and minifies some parts of the HTML (all of this done in the cached template, so runtime speed is not affected.)  Initial impressions seem like it's mostly a waste of time (at least in my inline script tags, which are not common), but offers a small benefit.  I haven't been messing with it much since.
Sounds like fun to me, though. Then again I'm a sucker for minification... (Except for the odd bug like the one I hacked into fixed yesterday.)

Re: JavaScript caching
« Reply #21, on July 5th, 2011, 07:08 PM »
I haven't got a lot to add except...
Quote
But it would mean having to decode after that, wouldn't it...?
I don't even remember seeing a ready-made function for that in ECMAScript...
json_encode on the PHP side, exports raw JS code that you can use... JSON is, after all, just JavaScript expressing an object.

Re: JavaScript caching
« Reply #22, on July 5th, 2011, 08:10 PM »
Ah... Yes indeed. That makes sense :lol:

I just gave it a try. It's interesting, but I don't know if it's worth replacing JSE with it.
For instance it'll always escape "/", even when we don't need it. It'll also turn tabs and newlines into \t and \n with no way to disable these. And re-deescaping them could prove complicated (I don't remember if regexp in JS supports assertions... Because without an assertion, you can hardly turn \\t into a litteral \t, it'll just turn it into \ followed with a tab...)

Re: JavaScript caching
« Reply #23, on July 5th, 2011, 10:53 PM »
With PHP 5.3 json_encode has parameters you can pass. One of them, JSON_UNESCAPED_SLASHES, might be relevant, but it's undocumented (yay)

It should be faster, though.

Re: JavaScript caching
« Reply #24, on July 5th, 2011, 11:35 PM »
It's actually documented but not in the json_encode help page... You have to go look into the JSON module and search for 'resources' or something like that. I found it originally via Google (duh)...

I posted *after* looking into these. At that point I already knew that none of the params (including the 5.4.0 params) could help in this situation.

Re: JavaScript caching
« Reply #25, on July 5th, 2011, 11:38 PM »
Quote
It's actually documented but not in the json_encode help page... You have to go look into the JSON module and search for 'resources' or something like that. I found it originally via Google (duh)...
Bah. That's retarded. At least some of the parameters are implied through example, this one isn't. I wasn't sure what it did, so wasn't sure if it'd help or not.

Re: JavaScript caching
« Reply #26, on July 6th, 2011, 12:01 AM »
Just Google it ;)

I suppose strtr() isn't that slow anyway...

Tss, another day of frustration in my code. Among many other things, I wanted to set the action buttons straight in the post list. I ended up settling for the 'easy' solution -- display:table. Unfortunately I don't have Safari installed right now. Since it's not very table-friendly I'm afraid it'll fail. I tested under Firefox 5 and it doesn't work very well already... Only Opera 11 loves it, meh. And, strangely, IE9... Although it doesn't set .quickbuttons to full height (it should fill the table... Like any NORMAL table cell...!)

I'm getting very annoyed by the constant changes in my CSS. I'd really like to call it quits and commit it, but I'm scared most people will think it doesn't look as good as a stock SMF2, ah ah...
Heck, I made many changes to the visuals this week really. Even the windowbg divs have finally lost their border... (I'm not kidding. I was adamant this border would be in the final design... Until I decided it wasn't needed.)

Re: JavaScript caching
« Reply #27, on July 6th, 2011, 12:09 AM »
Quote
I'd really like to call it quits and commit it, but I'm scared most people will think it doesn't look as good as a stock SMF2, ah ah...
You know the answer to that, screenshots! You might think it may not be as good as a stock SMF 2, and it may not be. It may be better, but right now the only person who knows for sure is you...

Re: JavaScript caching
« Reply #28, on July 6th, 2011, 09:31 AM »
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
I know, but it's there, so... I just use it. It's a matter of being consistent with the existing codebase. There would be too much to change otherwise.
Also, sometimes it can be helpful to know what the *intended* purpose of a variable is. I know it has helped me... Don't forget that our code needs to be readable by future devs!
I didn't say using any form of hungarian notation was wrong.  Using "userTitle" for user input and "title" for internal makes sense.  Mozilla uses a convention of "aTitle" for arguments.

But, even for a strictly typed language, type-based hungarian notation seems silly and wrong to me.  All the more for a loosely typed one.

But, just IMHO.  I know there's tons of programmers out there with different opinions.
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
You mean like <base href>...?
(man this auto strip quotes thing is kinda confusing.)

No, just a:

var smf_base_url = "http://wedge.org/";

In my experience, Google doesn't read bare links after that, e.g.:

var smf_theme_url = smf_base_url + "/Themes/default";

Obviously, it becomes more complex when those urls aren't always in the base - as with SMF, and as with some of my stuff.  I think I tried separating with + without any luck, and ended up using a htaccess forbidden rule or something (because I didn't want the separate content url indexed.)

There are issues with base href in some browsers iirc.  It generally works and I've used it before, but I know sometimes it doesn't work like you'd expect.
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
I don't even remember seeing a ready-made function for that in ECMAScript...
In TOX-G:

var x = <tpl:json value="{$x}" />;

Note that you have to wrap that in CDATA (e.g. around the whole script, as often done in SMF for XHTML compat) or use <tpl:container doctype="html5"> if you want it to be escaped right in <script> (which has text semantics.)  In attributes (e.g. onclick="blah(<tpl:json value="{$x}" />);") it will escape the html entities as well, which is necessary to avoid security holes.
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
It seems to be the case for me... In which case, why did JSE ever begin to escape them...?
I don't know, that function was created after I left.  There are valid reasons for escaping js in many ways to avoid security holes, but it seems like it does too much.
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
Gzip doesn't really matter here, you'd be surprised with the strange results I've been getting. (There's a recent discussion somewhere about the effort I went into converting some variable names to be one char instead of long stuff, and in the end the resulting gzipped file was actually longer by a few bytes... I just couldn't believe it.)
Keep in mind that repetition is what's key to compression.  Consider the following:

Code: [Select]
function show()
{
   var should_i_really_show_it = !this.is_showing();

   if (should_i_really_show_it)
      this.really_show_it();
}

function hide()
{
   var maybe_really_hide_it = this.is_showing();

   if (maybe_really_hide_it)
      this.really_hide_it();
}

It might seem like the variable names are way too long, and they are (from a this-code-is-ugly perspective.)  But really, the biggest problem for compression is that it isn't very repetitive.  Consider instead:

Code: [Select]
function show()
{
   var is_it_showing_now = this.is_showing();

   if (!is_it_showing_now)
      this.really_show_it();
}

function hide()
{
   var is_it_showing_now = this.is_showing();

   if (is_it_showing_now)
      this.really_hide_it();
}

Yes, the variables are shorter.  But, the more common pattern may compress better.

One of the important things minifiers do is create patterns.  For example, doing "if (x) {} else y();" may actually be better than "if (!x) y();" in a long document that uses "if (x) {" a lot and uses "} else" a lot.

Potentially, but not necessarily.  Without actually testing, the cases can always be strange, and small scale experiments are rarely accurate.

Anyway, I was guessing that using \ns might gzip better, at least because the pattern after the \n and including it may be more likely to match (such as "\n     <div".)  Obviously, the \ does muck this up and so it might not really compress better after all.  Certainly it's the same number of bytes whether a newline or n is escaped.
Quote from Nao/Gilles on July 5th, 2011, 07:01 PM
Ah, would be swell if Apache added support for 7z or even RAR... :lol:
Well, recall that the iPhone 3GS was able to load pages much faster than the 3G.  This wasn't because they gimped the network for the 3G, but because the processor was more capable to deal with it.  Engines like 7z, rar, and bzip2 achieve better compression ratios but at the cost of more cpu, which can be a problem (if dynamic) on the server side, and can of course be a problem on the client side (e.g. for mobile.)

Anyway:

https://bugzilla.mozilla.org/show_bug.cgi?id=173044
https://bugzilla.mozilla.org/show_bug.cgi?id=366559

AFAIK rar is encumbered so I wouldn't hold my breath for it.
Quote from Nao/Gilles on July 6th, 2011, 12:01 AM
Just Google it ;)

I suppose strtr() isn't that slow anyway...
Actually, it is.  On long strings, str_replace can take only 15% the time, and on short ones 30%.  The situation was reversed in PHP 4, which is why I used strtr a lot in SMF (I also prefer its syntax.)  In fact, in some cases preg_replace may beat strtr, I suppose, based on my benchmarks.

I think that in that PHP fork that was posted here, the guy sped up strtr.  Clearly there's room for improvement.

By the way, json_encode, on my setup, is in fact even faster than addcslashes (which by the way, is usually faster than addslashes, even with the same character list.)  Haven't checked its memory impact, though, would probably need to xhprof that.
Quote from Nao/Gilles on July 6th, 2011, 12:01 AM
I'm getting very annoyed by the constant changes in my CSS. I'd really like to call it quits and commit it, but I'm scared most people will think it doesn't look as good as a stock SMF2, ah ah...
Heck, I made many changes to the visuals this week really. Even the windowbg divs have finally lost their border... (I'm not kidding. I was adamant this border would be in the final design... Until I decided it wasn't needed.)
Ehh... I suck at design which is why I left the HTML and CSS stuff mostly alone in SMF.  The nice thing about a versioning system like svn is you can always svn merge -c -123 and undo a commit.

-[Unknown]

Re: JavaScript caching
« Reply #29, on July 15th, 2011, 10:55 AM »
I need to answer that post at some point..... :^^;:

Just a quick example on the issues of gzip compression.

index.css, background image for the selected menu item.

Code: [Select]
h4:hover extends .arrows, h4.hove extends .arrows
background-color: #5a6c85
background-position: 98% -28px

Given that .arrows is defined as such:

Code: [Select]
.arrows
background-image: url($images/menu.gif)
background-repeat: no-repeat

Now, this css file weighs in at 74.548 bytes.

Replace the first block with this:

Code: [Select]
h4:hover, h4.hove
background: #5a6c85 url($images/menu.gif) no-repeat 98% -28px

This will duplicate menu.gif (we're talking about base64 encoding, so we get the actual data twice). That particular file is 150 bytes.
The resulting index.css is 74.721 bytes, i.e. 173 bytes more, which makes sense.

Now, let's gzip both files...
Results:
First file: 74.548 -> 30.432 bytes
Second file: 74.721 -> 30.430 bytes

So, the question would be: what's best, a short CSS file, or a short gzipped CSS file?
Obviously it depends on what your users are receiving. If their client can't receive gzipped data, they're better off with the first file. Otherwise, the second file is best.
As it happens, the difference is too small (2 bytes) to justify thinking too much about it. It's a very slightly noticeable improvement for a very, very small share of the community, and a negligible negative difference for the larger majority. I'll take the first version because, really, I like my inheritance system.

But it's a real annoyance that I can't find a way to improve gzipping on the first test case...

Re: JavaScript caching
« Reply #30, on July 16th, 2011, 08:46 AM »
Quote from Nao/Gilles on July 15th, 2011, 10:55 AM
The resulting index.css is 74.721 bytes, i.e. 173 bytes more, which makes sense.

Now, let's gzip both files...
Results:
First file: 74.548 -> 30.432 bytes
Second file: 74.721 -> 30.430 bytes
Well, again, it doesn't always necessarily follow logic.  But, internally, it is building a dictionary.

So even though it's less bytes, it may not fit a global pattern.  How often in the entire CSS file do you use the shorthand versions?  Probably not often.

Maybe you use:

"background-repeat: no-repeat;
background-color: #5a6c85;
background-position:"

A lot.  But you rarely if ever use:

"background: #5a6c85 url($images/menu.gif) no-repeat"

That sounds likely.  So, with the first one, gzip is able to build a larger dictionary entry, which gets used more often, saving more bytes.  It's not like gzip compresses x% always, it just compresses patterns like any other compressor.

That said, less bytes may be better for the client than less gzipped bytes, in some cases.  Especially for the browser and RAM usage.  But generally, less gzipped bytes is probably better.  In this case, yes, obviously, doesn't matter.  They probably both fit into the same number of packets anyway.

Are you aware of any clients (aside from poorly written HTTP clients in PHP and some robots) that can't handle gzip?

-[Unknown]