CSS and JavaScript minification

Nao

  • Dadman with a boy
  • Posts: 16,079
CSS and JavaScript minification
« on May 10th, 2011, 06:43 PM »
Feature: CSS and JavaScript minification
Developer: Nao
Target: everyone
Status: 100% (believed to be complete.)
Comment:

Wedge likes bandwidth optimization. Wedge is sad when its pages don't show up immediately. Wedge is a good boy.
Whenever you upload a modified CSS/JS file to your styles or scripts folder, it will catch it, minify it on the fly and cache it.
Minification is a process that removes all comments, whitespace and things that aren't needed for the code to work. CSS is easy to minify, but JS is sometimes a bit capricious, so I added support for JSMin minification (the safe one), Packer 3.0 (the efficient one), and no minification at all (useful for debugging your scripts.)

We recommend using Packer, it's nearly as efficient as UglifyJS. A note to modders: just make sure your JavaScript passes JSLint validation, or at least you don't forget to add semi-colons at the end of var fn = function () { ... }; declarations, otherwise Packer will choke on it (it's fixable, but we'd rather not make the script slower because of lazy programmers.)

Re: CSS and JavaScript minification
« Reply #1, on June 19th, 2011, 10:32 AM »
How is this implemented?  It would be nice if it does it based on e.g. a production or non-debug setting.

I did some post-deflate tests, and in fact for many projects am using a script that uses Closure Compiler, uglify-js, packer (although packer2, need to either fix packer3 PHP or write some hookup for offline automation... maybe node.js?), and YUI.  Usually uglify-js is the winner.

For CSS, I've not been impressed with a PHP css compressor I found that was popular.  YUI seems to do a lot of nice things, and do pretty well after deflate without breaking things.  That said, I've been looking at lesscss/sassy lately, which supposedly pack as well, and haven't compared them yet.

Closure Compiler actually returns errors (and can auto-correct most of them) for things like the automatic semicolon insertion issue.  That's why I use it as well, and report errors it reports.

-[Unknown]

Re: CSS and JavaScript minification
« Reply #2, on June 19th, 2011, 06:45 PM »
I believe it's Packer 3.1 with a few changes off the top of my head. It's done by examining the timestamp of the files and comparing it to the last-built one, so if you change a file, it also updates the cached version.

It can be disabled in the admin panel.

Re: CSS and JavaScript minification
« Reply #3, on June 20th, 2011, 12:37 AM »
Yes, it's based off Packer 3.1. Dean Edwards has been working on Packer 4.0, though, and he told me he'd be putting the PHP version online soon. He's sort of hoping I'll be taking over the PHP version's maintenance but my knowledge of base2 (especially its PHP version) is rather limited. Anyway-- I'm interesting in giving 4.0 a shot and using it if it helps save even a few bytes off the 3.1 size.

I don't know of any reason why Packer should require 'offline automation'. Granted, the PHP version doesn't work out of the box, but it only took me a couple of hours to turn it into a working version (it really doesn't have THAT many bugs. It's just that Dean lost interest in the PHP version.)

Regarding semicolons, I provided a fix in the source file, but I commented it out because I think it's best to educate devs into using semicolons as much as possible. That's why I said it's fixable -- anyone's free to uncomment the fix, and then test their broken files. If that fixes them, then it means they should be careful about semicolon placement.

PS: would you like to get svn access to the files? Maybe it'd be easier for you if you could discuss the features without having to resort to guess work ;)

Re: CSS and JavaScript minification
« Reply #4, on June 20th, 2011, 01:47 AM »
Quote from Nao/Gilles on June 20th, 2011, 12:37 AM
Yes, it's based off Packer 3.1. Dean Edwards has been working on Packer 4.0, though, and he told me he'd be putting the PHP version online soon.
Sure, I'd like access, if only for this.  I considered spending the time to fix the PHP base2 and stuff, and even fixed it some, but I figured I'd check with him to see if it was a known issue he was fixing.  Never got a reply, so I left it alone.
Quote from Nao/Gilles on June 20th, 2011, 12:37 AM
Regarding semicolons, I provided a fix in the source file, but I commented it out because I think it's best to educate devs into using semicolons as much as possible.
I agree; keep it without the fix.  But report an error message with file and line info.  I also prefer this for trailing commas, since I want to be able to dev against the unpacked js. even in IE.

I definitely wouldn't want it to just pack my css or js every page view. Not only for the cost of the mtime IO checks, but also because it makes debugging harder by far.  I don't know if you do js debugging, but I definitely do, and debugging packed js is similar to debugging an optimized exe with no debug info.

That's why I'd suggest a dev mode switch.  Also because, it means you can have the dev mode only apply to administrators (with a conspicuous message, like when you leave upgrade.php uploaded), such that you can "stage" js and tox changes, test them, and then "push them live" for everyone with a click of a button.

Also, are css/js files served with the overhead of PHP?  I can show benchmarks that even on nginx/php-fpm, this overhead is not nothing.  I suggest providing some way to avoid if possible, as Apache and nginx both have deflate mechanisms, and its support is detectable.

Oh, and I automate everything I can.  Take a look at my TOX-G makefile - I don't leave it to chance that I forget the copyright year, or to run the tests, or etc.  And I have packing, optipng, etc. all automated as well.  If I can remove the human component, I do.

-[Unknown]

Re: CSS and JavaScript minification
« Reply #5, on June 20th, 2011, 02:31 AM »
It's not even a dev mode switch, it's right there in Server Settings to disable it, or to switch packers, since MinJS is available too, but I think Packer generates smaller files. It's just enabled by default.

The files are not served with a PHP overhead, the only overheads incurred generally are the mtime checks to see if things have been modified, and that's used to flesh out the request URL (and filename) where the style can be found.

Hang on while I sort out SVN access, I'll PM you with the details shortly.

Re: CSS and JavaScript minification
« Reply #6, on January 30th, 2013, 12:59 PM »
(Oh, a topic with a post from [Unknown]... :P)

Nice little time waster on my side: yesterday I discovered that Google released (years ago!) an online tool for compressing JS through Closure on their website.

Because Closure's results are pretty much better than UglifyJS's (UglifyJS is only better on jQuery, for some reason... they even acknowledge it on their official website :^^;:), albeit much slower to compile, I'd always wanted to compare with Packer.

And here I go... I implemented a 'closure' minification method for Wedge.
It works, but it's definitely not going to be default!

- It's SLOW AS HELL. script.js takes about 5 seconds to compile, for instance! It takes like half a second with my local Packer. And I'm not even checking... It may be less.

- Google has set a limit on the number of times you can compile a script per hour or something. While I guess it's perfectly acceptable in a situation where you would simply upload modified versions of your scripts from time to time, you definitely can't debug anything with that. (Not that you'd want to -- see compilation time.) I need to do the debugging on THAT particular platform though, and believe me I'm enraged that they're not providing me with a way to bypass their limitation WHILE I'm implementing support for THEIR product... -_- Overall, it seems like they can 'take' a dozen compiles at most, and then... Poof. I haven't been able to compress anything this morning, for instance.

- I implemented a fallback -- if Google fails, then Wedge will just re-do the job with good ol' Packer, and skip Closure for all subsequent minification calls in the same page load.

- It's more efficient than Packer, saving a lot more bytes in uncompressed form, but once gzipped, it's pretty similar. On average, I'd say that a Closure version is about 100 bytes shorter for each gzipped file. I'm thrilled with the extra bytes, you know how I am, but I'm definitely not thrilled enough in a realistic configuration.

So, it'll be an option I think, but probably no more than a non-default, warning-filled option...


Re: CSS and JavaScript minification
« Reply #9, on January 30th, 2013, 04:20 PM »
I've only called the API 5 or 6 times since this morning (i.e. about 7 hours ago.)
At all times, Google returned an "Error(22): Too many compiles performed recently. Try again later."
Their help site mentions having to wait for "an hour" before the quota gets reset, or something.
Yeah, sure.

If I can't do my tests, I can't implement it. Too bad, the code was working and all yesterday... :^^;:

I'll probably commit it to the attic, so that way if someone wants to re-use it later, they can. (e.g. Google fixes their strict policy.)
Re: CSS and JavaScript minification
« Reply #10, on January 30th, 2013, 05:23 PM »
And, of course, compilation started working again about 10 minutes after I posted that...
I decided to work on zoomedia.js rather than script.js for this, it's shorter, maybe Google will be cooler with me. I simply moved the text strings over to it.

I've devised a relatively complicated preg_replace to go from \u1234 in the returned text to some actual UTF8 chars. What a mess, really...

Still -- now it seems to be working. I'd appreciate if people could test a full wipe of their JS cache folder with Closure activated, once they install the updated Wedge.

Re: CSS and JavaScript minification
« Reply #11, on January 30th, 2013, 09:24 PM »
I get the feeling it is a nice idea but not actually worth the hassle. I remember having to help debug an Impact JS game a bit back that was compiled with Closure that didn't work any more, because there was a set-up involving (x - -k) or something in the original source that gets compressed to x--k which is of course no longer a valid expression.[1]

100 bytes is 100 bytes, sure, but at a massively increased debug and maintenance cycle? I would leave it alone personally.
 1. I only vaguely remember the circumstances, it was a 3rd party library that wasn't really designed to be minified too heavily.

Re: CSS and JavaScript minification
« Reply #12, on January 31st, 2013, 10:49 AM »
The main problem right now is that I've rewritten a few things not directly related to Closure, that I'd like to commit still...
For instance, I've discovered that Closure's very very strange decision to convert all UTF chars to \u00xx is actually better for gzipping, so I rewrote @language handling to force including \u chars instead. Which in turn pushed me to write a 'generic' function for that in Class-String. Which pushed me to write the opposite function, etc... See the pattern?

I'll just make sure that the code for Closure can be reverted without too much hassle, but I'll still commit it, and just recommend against using it.

BTW, I think the 'x - -k' bug was also in earlier versions of Packer or something... Maybe that's where you heard about it..?

Re: CSS and JavaScript minification
« Reply #13, on January 31st, 2013, 04:44 PM »
Actually I first encountered it in JSMin, but the principle tends to hold ;)

It's always good to have such facilities :)