I thought I'd share some of my frustrations with you all.
I won't get into the detail of it but the last few days I've been snowed under - as I mentioned briefly - in building an app using Node.js, Express, Socket.io and MongoDB. I'm not going to explain what the app is, hopefully you'll see soon enough, but it's certainly been an interesting experience for me so far, and I feel like sharing what I've found.
First up, for those of who aren't familiar with Node.js, let me explain. We here are far more used to deploying apps where we have a web server like Apache, which receives requests, passes some of them onto PHP to process the contents of the page, and handles the page until it's ready and sends it back to the user.
It's convenient, practical and we're all used to it, but the app I've been working on lately requires longlife connections, what we would normally consider to be Comet style. I could have done it in AJAX but Comet is far less vicious on the server, especially given the real-time nature of the app in question, but in Apache/PHP this just is not practical.
Enter, then, Node.js. At first glance it sounds fantastic - you're working in JavaScript on the server side (so it's easy to learn, in theory) but there's no webserver. In other words, you write some JS that also functions *AS* the webserver.
From the Node.js docs, this would be pretty much the simplest possible webserver in Node:
Code: [Select]
Yup, you see that right. You're handling raw HTTP requests and headers. For someone like me who's been behind the cosy framework of PHP for years, this is... something of a change of pace. It's not a bad one at all, it's just a lot to take on board.
Especially with the heavy-going view it has towards I/O. In PHP, the script runs from top to bottom, and it waits for I/O to complete - so execution of a given page waits while a DB query completes, for example. This doesn't happen in Node, or at least it shouldn't.
The idea, really, is that you push the request off to another function and handle its response in a callback, so that mainline execution can get back to whatever it was doing. This, for me, is a *major* change of pace because it's just not a mentality I'm used to working in.
But, so far so good, actually. I was able to fashion something workable with that, but it wasn't really what I wanted, I wanted something that did some of the work for me. Yes, that's right, I started looking at frameworks and ended up with Express.
Express makes a lot of the work easier, but it has two problems. The first is that its documentation is absolutely crap. No, really, it's ABSOLUTELY SHIT. http://expressjs.com/guide.html is the entire official guide.
There's several sub-problems. Firstly, it makes no mention of Express 3.0 at all and only barely mentions 2.x in passing for migration. It would be OK reading the source, if Express were all the source, except it isn't. It's a framework on top of another framework, called Connect, and it's very hard to figure out WTF is going on when Express doesn't seem to point to another about Connect, like any other documentation.
The second problem is probably a bigger one in the scheme of things. A lot of people did figure out how to use Express and muddle through. And some people have written blogs on it, others have asked and answered things on StackOverflow. The result was that I've had dozens and dozens of tabs open in my browser for the last couple of days as I start to piece things together.
I mean, I have a functioning webserver, it receives requests, if they're dynamic pages, they're routed properly. If they're static resources, again routed properly. 404 and 500 have their own proper pages with correct headers and everything. But it took me over a day to figure all this out to make it work how I wanted because the documentation is so poor. And I mentioned that I had dozens and dozens of tabs open... I still have about 18 tabs open for the next stage of the project, and that's the problem in itself: there's so many bits and pieces on it but they're all haphazard and many of them are legacy items.
For example, I found one tutorial from late 2010 which refers to using Express methods for handling static content, and it refers to staticGzip and staticProvider; the former indicating that content should be gzipped and the latter to indicate that a given path is where static content should be served from.
But the manual doesn't mention these, it only talks about a method simply called static, which has a slightly different calling argument structure to staticProvider but does mostly the same job. After 20 minutes of digging around I found that staticProvider was deprecated in favour of simply static[1] and that staticGzip is no longer directly available because most people deploying such apps would serve static content like that from a CDN and not from your own server. A reasonable if slightly misguided idea, I think. As it happens in this case I'm looking at deploying with sufficiently long-life expiries that it doesn't really matter so much.
But anyway, I've spent so long trying to piece everything together that it's just left me feeling so frustrated.
I haven't actually told you the worst part yet, actually. Node has a nifty-in-principle tool called npm for installing packages. There's a central repository that lists packages and most of the time you can do npm install <package> and boom, it'll download. Unless it's Express, in which case you have to have make installed to be able to install it. This took me a while to figure out due to no-one describing it anywhere and npm giving me less than helpful messages.
It gets a bit better, actually. You can declare a package.json file which indicates a package's dependencies. This can be the Node versions supported, or it can be the version(s) of packages you need. My project needs Express, Socket.io and MongoDB's connector, seems straightforward enough. Until you hit the joys of figuring out which versions you need, of course.
Oh, and there's more fun. Each of those has other dependencies which also needs to be met. And it's not clear whether some of those come installed or as optional extras, Express for example states a 'devDependency' of Jade, though it doesn't actually install Jade unless you ask it to. Again, not documented anywhere. Then more version juggling.
Jade is actually pretty neat, though. Had we not had the template skeleton I might have suggested a move to it, because it allows for replacing blocks of a document, for prepending/appending content to blocks and so on. Plus the fact that Jade's syntax is incredibly tight and descriptive and not the usual verbosity of HTML itself. http://jade-lang.com/ if you're curious.
Anyway, that's enough rambling and venting from me. For my next challenge, figure out how sessions work (and figure out how to make them compliant with the EU laws, ahahahahah) and then bolt on Websockets type connections to it and maybe see if we can't do something truly wonderful from there on in!
The bottom line is that Node and its tools make for a very interesting and flexible platform for deploying unconventional applications but the lack of good documentation for major components (Node itself is well documented, just not its good bolt-ons so much) really makes it a struggle to implement anything.
I won't get into the detail of it but the last few days I've been snowed under - as I mentioned briefly - in building an app using Node.js, Express, Socket.io and MongoDB. I'm not going to explain what the app is, hopefully you'll see soon enough, but it's certainly been an interesting experience for me so far, and I feel like sharing what I've found.
First up, for those of who aren't familiar with Node.js, let me explain. We here are far more used to deploying apps where we have a web server like Apache, which receives requests, passes some of them onto PHP to process the contents of the page, and handles the page until it's ready and sends it back to the user.
It's convenient, practical and we're all used to it, but the app I've been working on lately requires longlife connections, what we would normally consider to be Comet style. I could have done it in AJAX but Comet is far less vicious on the server, especially given the real-time nature of the app in question, but in Apache/PHP this just is not practical.
Enter, then, Node.js. At first glance it sounds fantastic - you're working in JavaScript on the server side (so it's easy to learn, in theory) but there's no webserver. In other words, you write some JS that also functions *AS* the webserver.
From the Node.js docs, this would be pretty much the simplest possible webserver in Node:
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');
Yup, you see that right. You're handling raw HTTP requests and headers. For someone like me who's been behind the cosy framework of PHP for years, this is... something of a change of pace. It's not a bad one at all, it's just a lot to take on board.
Especially with the heavy-going view it has towards I/O. In PHP, the script runs from top to bottom, and it waits for I/O to complete - so execution of a given page waits while a DB query completes, for example. This doesn't happen in Node, or at least it shouldn't.
The idea, really, is that you push the request off to another function and handle its response in a callback, so that mainline execution can get back to whatever it was doing. This, for me, is a *major* change of pace because it's just not a mentality I'm used to working in.
But, so far so good, actually. I was able to fashion something workable with that, but it wasn't really what I wanted, I wanted something that did some of the work for me. Yes, that's right, I started looking at frameworks and ended up with Express.
Express makes a lot of the work easier, but it has two problems. The first is that its documentation is absolutely crap. No, really, it's ABSOLUTELY SHIT. http://expressjs.com/guide.html is the entire official guide.
There's several sub-problems. Firstly, it makes no mention of Express 3.0 at all and only barely mentions 2.x in passing for migration. It would be OK reading the source, if Express were all the source, except it isn't. It's a framework on top of another framework, called Connect, and it's very hard to figure out WTF is going on when Express doesn't seem to point to another about Connect, like any other documentation.
The second problem is probably a bigger one in the scheme of things. A lot of people did figure out how to use Express and muddle through. And some people have written blogs on it, others have asked and answered things on StackOverflow. The result was that I've had dozens and dozens of tabs open in my browser for the last couple of days as I start to piece things together.
I mean, I have a functioning webserver, it receives requests, if they're dynamic pages, they're routed properly. If they're static resources, again routed properly. 404 and 500 have their own proper pages with correct headers and everything. But it took me over a day to figure all this out to make it work how I wanted because the documentation is so poor. And I mentioned that I had dozens and dozens of tabs open... I still have about 18 tabs open for the next stage of the project, and that's the problem in itself: there's so many bits and pieces on it but they're all haphazard and many of them are legacy items.
For example, I found one tutorial from late 2010 which refers to using Express methods for handling static content, and it refers to staticGzip and staticProvider; the former indicating that content should be gzipped and the latter to indicate that a given path is where static content should be served from.
But the manual doesn't mention these, it only talks about a method simply called static, which has a slightly different calling argument structure to staticProvider but does mostly the same job. After 20 minutes of digging around I found that staticProvider was deprecated in favour of simply static[1] and that staticGzip is no longer directly available because most people deploying such apps would serve static content like that from a CDN and not from your own server. A reasonable if slightly misguided idea, I think. As it happens in this case I'm looking at deploying with sufficiently long-life expiries that it doesn't really matter so much.
But anyway, I've spent so long trying to piece everything together that it's just left me feeling so frustrated.
I haven't actually told you the worst part yet, actually. Node has a nifty-in-principle tool called npm for installing packages. There's a central repository that lists packages and most of the time you can do npm install <package> and boom, it'll download. Unless it's Express, in which case you have to have make installed to be able to install it. This took me a while to figure out due to no-one describing it anywhere and npm giving me less than helpful messages.
It gets a bit better, actually. You can declare a package.json file which indicates a package's dependencies. This can be the Node versions supported, or it can be the version(s) of packages you need. My project needs Express, Socket.io and MongoDB's connector, seems straightforward enough. Until you hit the joys of figuring out which versions you need, of course.
Oh, and there's more fun. Each of those has other dependencies which also needs to be met. And it's not clear whether some of those come installed or as optional extras, Express for example states a 'devDependency' of Jade, though it doesn't actually install Jade unless you ask it to. Again, not documented anywhere. Then more version juggling.
Jade is actually pretty neat, though. Had we not had the template skeleton I might have suggested a move to it, because it allows for replacing blocks of a document, for prepending/appending content to blocks and so on. Plus the fact that Jade's syntax is incredibly tight and descriptive and not the usual verbosity of HTML itself. http://jade-lang.com/ if you're curious.
Anyway, that's enough rambling and venting from me. For my next challenge, figure out how sessions work (and figure out how to make them compliant with the EU laws, ahahahahah) and then bolt on Websockets type connections to it and maybe see if we can't do something truly wonderful from there on in!
The bottom line is that Node and its tools make for a very interesting and flexible platform for deploying unconventional applications but the lack of good documentation for major components (Node itself is well documented, just not its good bolt-ons so much) really makes it a struggle to implement anything.
1. | Including a snide mention from Express's author that JSLint is 'lame' because it can't differentiate between 'static' being used as a method name as being used to indicate something being static. |