Good to find you post SXSW, it was really good to talk to you and I definitely remember this conversation. To a lot of the people who are being very critical, it’s important to remember that the comment “you’re never going to be big enough to worry about it” is true for a majority of companies. That doesn’t mean write bad code, and it doesn’t mean pick a framework that will be difficult to divorce yourself from later. It does mean when you are trying to demonstrate business value, you probably won’t be at a size where scaling is a concern and you should be focusing on the “demonstrating value” part.

Once you’ve demonstrated enough value, and once you’ve enough traffic that scaling will be a concern, you’ll probably (1) know where your pain points are and (2) have gained enough experience to begin tackling them. Otherwise, you will be taking your best guesses at where your scaling won’t work and end up with the Premature Optimization Anti-Pattern.

The original quote “frameworks don’t scale, architectures do” is attributed Cal Henderson (though Blaine was the one to first mention it at our panel).

But again, almost all websites won’t even need a second server so thinking about issues like this right off the bat is careless, IMO.

At Qrimp, http://www.qrimp.com, we use Mosso to scale the infrastructure so we can concentrate on the developing the software. Mosso abstracts the hardware layer into a multipurpose infrastructure and Qrimp abstracts the software into a multipurpose platform. The benefits are compounded.

These simplifications of layers are going to continue to feed Moore’s law. While Moore focused on hardware computing power, but his ideas can be extrapolated into solving problems with technology. Not only is the hardware getting faster, but it’s getting easier to put it to use, because building the software is getting easier. It’s exponential growth in both directions.

Seriously… its actually pretty easy to scale now — in some cases within a utility model.

Just got back from beers with the guys from 3tera… so falling upon this now… is interesting.

3tera provides a superior virtual infrastructure IDE + core services & management… while ec2/appengine provide a functionally scaled down albeit more utility-ish service.

Most think only of scaling ‘up’ while gracefully scaling ‘down’ is probably more important, because as you point out, most sites won’t need the upper capacity of a given container. Using as little as possible and being charged only for that use… is scale… too.

bah… scale is for everyone now.

I’ve been pushing them on the fact that even though their service is so superior, the cost of entry into their realm is too prohibitive for the OSS world. I think if some love and support were there, I bet some initial subsidization could occur to bring down the initial costs for OSS related projects. 3tera seems very open to this. For disclosure, I am a customer and a friend of the co… I would just really like to see general grid capacity behind their tech.

…this is like ‘real names’ stuff :D

enjoy!

(;||<

I remember when I first started on my PHP Framework (CX) that I read something along the lines of “if you had anything worth saying, you wouldn’t be building a framework to blog on - you would be blogging!” And like your point above, you just need to get out and do it.

But, a “job worth doing is worth doing right”. Big companies didn’t make it without some planning.

One of the senior devs in my company was gathering some stats the other day and he noticed we’re averaging about 2.25 million page views a day on our multiple listing app. Something like 8+ million hits a day. Scalability with this app does have a lot to do with the framework (which we’re seriously lacking), but there are also more practical things like simply setting up a proper robots.txt because search engines alone in one day would kill our servers.

I don’t think most people have a clue about real scalability issues, so I tend to agree with you in that you just don’t have to worry about it because no site that you or I make is going to get this big. Period. You’re never gonna need three photo servers, a couple database servers, load balancers, and so forth. You’re never gonna need to worry if that one text file you call on every page is going to build up the bandwidth and be a performance issue, or whether you need to start caching crap left and right. Frameworks are good but generally won’t help or hinder the real issues.

Good point! Extreme scaling (disregarding the digg-effect) is the domain of only a few thousand sites on the web (ok I’m estimating but it can’t be a very large percentage). There was a lot of chat earlier this year about the scaling capabilities of RoR, Django, and other new frameworks, and about the possibilities of Amazon’s cloud computing service being a potential solution.

For any new site it’s a question of balance I suppose when deciding where the dev resources are allocated (product vs optimisation/maintainability/scaling). I’m not clued-up enough on Pownce to know if it is a success or not, but it sounds like even Kevin Roses’ reputation is pulling in the punters.. yet. On one hand you have to show enough product to get a buzz going (say 37s and Getting Real), but if it takes off then you need to make sure the site won’t fall over!

Ultimately it might just come down to money, how many devs you can afford (and what quality), and how much hardware you can harness.

Thanks for putting it as I had intended :) Indeed, as my first paragraph mentions, I’m referring to the scaling in reference to the points beyond “normal” scaling — the points at which frameworks start to fail, and even good development practices can cause problems.

These kinds of scaling problems are often argued about on the web, but rarely encountered. For example, people are quick to blame Rails in twitter’s scaling problems, but neglect to realize that twitter’s scaling problems are not a common problem. Worrying about the problems twitter has encountered during initial development would have been foolish. Had they spent the time worrying about it, they may have run into the same problem Pownce is having now: too much pre-planning, not enough releasing.

I would have thought if you’re writing code that is abstracted and loosely coupled with design patterns/OOP, and unit tested, along with other software engineering techniques, then, as Kyle mentions, when and if you hit scaling issues, introducing extra servers, slave db’s, daemons and all the rest of it should be easier than if you don’t use any of the aforementioned methods. I agree with his point that hundreds or even thousands of man hours up front to solve non-existent scaling problems are a mis-management of precious development time… time that could have been put into the product at hand. This is obviously dependant on the type of project you’re working on, expected traffic, and media you might be serving (i.e. video might require the use of a CDN or dedicated servers).

All in all mature frameworks most likely have the capacity to scale easily to a certain point, and good developers will mitigate some of the issues by using good software development practices. If your site really is large enough require custom scaling solutions, that’s when specialists can come in and do their thing.