Seeking Work (Remote). We built http://whisper.sh (website, iOS app, Server). We've built iOS apps for MTV, VH1, and others. I don't do a lot of business development; I just get on HN when we're done with a project.
Our team is small and we typically have one big project at a time. Specialties are a) iOS, b) Rails, c) Erlang.
FYI, the price doesn't seem to matter. A bid for $.50 isn't being fulfilled either. I still don't understand why a bid higher than the going rate isn't being fulfilled. :-/
attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over the past two years?
There once was a really nice quote here on HN: "you can't outsource responsibility".
If in two years time you've never ever had a look at what kernel you are running, especially while tuning a system for performance you only have yourself to blame.
Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it. Also, in your resources list you should have the exact machine configuration, there are tools to retrieve that sort of info automatically.
It's typical that the people at rackspace would simply drop in the requested hardware, and that you yourself deal with the configuration.
The smart money is on running some tests after they've done that to make sure it went ok. Asking for a CPU upgrade and not checking if they're operational is just plain stupid.
I figure you literally asked rackspace to upgrade the CPU, and that's what they did.
Did you explicitly ask them to install an SMP kernel with a specific version and they didn't do it ? Or did you expect them to do it but you didn't check if they actually did until today ?
Two full years of trying to tune a box for performance and not noticing this, then publicly blaming rackspace is simply cheap, an attempt at pinning the blame on rackspace, for something that you should have noticed long ago yourself.
Kudos for writing about it but the title should be "How I messed up". That's taking responsibility and then make sure it never ever happens again.
'Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it.'
Not sure why you've got 'stock' in quotes. Vendor kernels are used by hundres of thousands of servers, each sharing the same bug reports and security updates. There's a massive benefit unless you think you can do those bug reports and security updates better than your OS vendor.
Most custom compiles are by people who don't understand loadable modules or read somethign written before they existed.
Using a vendor-supplied kernel means that there are extremely likely to be other people using most of the same stack as you, many of them on the same hardware. If there are problems, it's likely that other people have noticed the issue, even if the bug hasn't been found, so it's much more likely to get fixed.
If you compile your own kernel (and/or copy of Apache, MySQL etc etc) you're running something unique to you. If you have problems, you're on your own.
If you're paying for Red Hat Enterprise, use the Red Hat Enterprise packages unless there's a good reason not to. If something goes wrong, you can call Red Hat support and have at least a steer in the right direction. Custom-compiling everything just for the sake of it, just to have new 'shiny' stuff is crazy.
It's not for the 'new shiny' at all, it's got to do with optimizing your kernel to match your hardware and getting rid of loadable module support in favor of a kernel that has on board exactly that which is needed to operate your system.
A 'stock' kernel has a whole pile of things in it that might be the next remote exploit, by removing such stuff you marginally increase security.
Other things you might need:
- kernel support for booting from raid filesystems without trickery
- processor family optimizations
- maximum number of cores (stock = 8, we run 16 on quite a few machines)
As for compiling, I do that anyway, it's a small job compared to the number of times that you need to do it. And you're just as much 'on your own' to solve problems, the chances of having them are less though (because the system you are running is considerably leaner).
Second your redhat enterprise solution, that's not what I'm using though on most of our machines (either centos or debian), but that's a good solution too.
Honestly your tone stings a bit, jacquesm. I guess i shouldn't have titled it 'don't trust rackspace' but my larger point is exactly the opposite of what you wrote - take responsibility for your servers - don't trust anyone, even the most expensive hosting provider, to do it for you.
That's a whole lot better, if that was the message then it somehow got lost to me.
Again, apologies for the tone, but it really seems to be a trend to make a mistake, 'blame someone', then blog about it.
What I would suggest you do, and this is meant very seriously, is find a cheaper hosting provider (EV1/The Planet is about half of what you pay right now) and spend the rest on getting a part-time sysadmin that really knows his stuff.
The difference in $ should be minimal, then look over the guys shoulder at how it is done, but keep doing what you know is your 'level' anyway. That way you get the best of both worlds, excellent care and you don't break the bank, at the same time you'll learn a huge amount.
And if your startup grows you just might have found an employee for the future.
Spend some time looking around, your best bet would be a guy or girl that does sysadmin duties for a larger company using UNIX that wants to make some extra $ in their spare time.
I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention.
"You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job.
Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.
I'm not a big fan of the tone, either, however, jacquesm is spot-on in his assessment.
For one thing, my understanding of Rackspace's business practices -- and I've only dealt with them peripherally, so I might be a bit wrong here -- is that they "manage" things like their network, and the actual server hardware, and stuff like that. So, if you want a CPU upgrade, sure, they'll do that. If you need your server rebooted, they'll do that too. But, they don't have anyone sitting there monitoring your system's performance metrics and doing your sysadmin duties for you.
The way I read it, Rackspace did do their job: they upgraded the hardware. It was up to the server admin -- not Rackspace -- to check that the software was then configured correctly.
And finally, I don't generally agree with statements of the form, "If you don't know X, you shouldn't be doing Y", but ... looking at dmesg and top are both really, really, really standard sysadmin operations. Entry level stuff, really. Sysadmin work doesn't just mean messing around with Apache's configuration; there are many more nuances, and it's likely that their system is vulnerable to problems that they don't even know about.
The tone is probably in large part because the OP does not take any responsibility for his own part in this and instead is pointing his finger at a third party that may have been partially at fault. But that is by no means sure.
This is typical with what I think is a real problem in society, the 'externalization of blame'.
Inability to see your own responsibility is a serious issue, and it is really pervasive. If I were in the OPs position I would be headbutting a piece of concrete for 20 minutes to make sure I never ever make a mistake like that again, and I would thank rackspace for finally finding the fault that I could have noticed in 5 minutes two years ago.
That's why you have post-delivery checklists, burn in tools and inventory management, staples of everybody that has # on machines that do customer work.
I'll try to keep my 'tone' better under control, apologies for that.
probably not a good idea to comment authoritatively on a company you haven't worked with, but no, kernel management is part of rackspace's job. performance and monitoring is part of their job. they have an SLA and this is absolutely part of it...
I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on.
As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff.
And if you outsource something you should at least know how to check up on the bits that you've outsourced.
Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'.
Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace.
All we get here is a pointing finger without any responsibility taken, that is not a realistic picture.
It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine").
Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem.
I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel."
How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ?
Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make.
But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.
"I said he should not be managing the servers their customers stuff runs on."
And what if it's only him? No go then huh?
You've been saying a lot of this kind of thing lately. That guy before with the App Store payment problem? You came down on him like a ton of bricks. And now this. Just because people haven't dotted every i and crossed every t. It's not exactly the hacker mentality is it?
Then he could hire a part-time sysadmin, there are plenty of those looking for work. I figure for $200 / month he can switch to a similar powered dedicated server with a competitor and pay a guy for 4 hours worth of real hands on sysadmin time every month. That way he pays roughly the same and comes out ahead in every way.
Managing UNIX systems that have to perform well under load takes quite a bit of knowledge. Sure, everybody can install 'ubuntu', 'redhat', 'gentoo' or whatever flavor is popular this week. But that does not make you a system administrator. I wouldn't trust myself with my customers machines either, simply because to stay up-to-date on all the holes in all the packages that you may have installed and keeping them patched is real work.
I don't think I came down on the app-store guy 'like a ton of bricks', in fact I gave what I thought was pretty sensible advice and offered (after Sam Odio did) to help him out.
But it's essentially the same problem as what is happening here, blame company X because of something that you caused yourself.
The app store guy:
- quit job before having money in the bank
- set up overly complicated corporate structure to avoid non-existent liability
This guy:
- take responsibility for a part of the operation that he's not qualified to do
- keep on messing for two years without calling in outside help (sure, it will cost)
And both of them point the finger at another party.
So maybe that's why it seems to you that this is a 'lot of this kind of thing'.
As for whether or not it is the hacker mentality is not my thing, I call it as I see it.
I've had people here rip me to bits for making a stupid remark (and rightly so), if you can dish it 'Rackspace is at fault because they don't know how to upgrade a cpu' or 'Apple is at fault because they don't pay me' then you should be able to take it.
It all comes down to what I paid for. Check your contracts carefully.
Some places basically just give you a computer and make sure that it always has power and network and that the hard drive is backed up, and everything after that is up to you. Other places give you 24 hour sys-admins that continuously monitor everything and basically manage every aspect of your server. The former obviously costs a lot less than the latter.
It's perfectly OK to outsource responsibility, but you've got to pay top dollar for someone to take on that responsibility. You cannot go for the cheap option and expect all the services offered by the expensive option.
on a separate application and server, yes, and it seems faster than apache/passenger, but it still suffers from the 'passenger choke' where touching tmp/restart.txt seems to pause all web traffic for 10-15 seconds, so we never considered it here...
wow. what really impresses me is mint isn't an engineering company - they're a marketing company that put some ajax on top of yodlee. yodlee's been around for over 10 years and they never built a brand or a community like mint.
mint is now #1 on my 'notes to self' when i get so focused on engineering my way to a solution i forget sometimes its' better to partner with a competitor and move on.
they outsourced the hard stuff (scraping/banking relationships) and then focused on building a product. nice work, brilliant execution.
I had no idea that Mint was a layer on top of another product. Can you explain this in greater detail? What is Yodlee? (I realize I'm proving your point by asking!) And what is the nature of Mint/Yodlee's relationship?
Yodlee is the service that took on the painful and expensive problem of actually building an engine that can securely store bank and brokerage passwords and use them to scrape hundreds of different financial sites.
They have a consumer front end. It's quite decent, actually, and it's free. Go to Yodlee.com and sign up. But they apparently make most of their money by (a) licensing their back end to services like Mint, and (b) serving as a kind of OEM for banks, who put up Yodlee's software rebranded as a part of the bank's website.
I have used Yodlee.com directly (and free) for two years; it has functioned extraordinarily well, and is invaluable--it saves hours of time dealing with personal finances, and provides very usable reports. I avoided Mint because I thought that Mint's very thin layer over Yodlee's services was apt to be an unstable business, and now we see that Mint users have been tossed to Intuit. I was also reluctant to expose all my financial details to Mint unnecessarily. Yodlee's own security credentials are high--good enough to convince just about all the large banks, brokerage houses, and credit issuers to trust them. Try a Yodlee account yourself, and you will see that Mint has been adding relatively little distinctive value over Yodlee's unique base.
That to me sounds like a better model than Mint. Sure Yodlee doesn't have the media buzz or "explosive" growth (Mint's growth didn't really impress me either.), but the infrastructure is extremely useful and will be around for a long time.
If Mint is doing well, Yodlee is too. If Mint fails, Yodless will still have a diverse client base to support its business.
But we don't know that Yodlee isn't doing 170M revenues a year either. Getting aquired and making money don't necessarily equate to each other in the most obvious ways.
Youtube was a huge money sink, and got a huge acquisition. Mint is, I believe at least, somewhat profitable, and only got $170M. The timing had a lot more to do with it than anything, but as a consumer, I'm a LOT more likely to give money to Mint than Youtube, and the YouTube founders are arguably a lot richer than the Mint founders.
So is this yet another reason to keep your web app basically free for the masses..?
$170 million in 2 years seems like reason enough for me.. but as Inaka points out: their execution was brilliant. So if you do have a brilliant app maybe you should keep it free.
Unlike most other webapps mint has a very lucrative revenue stream. They're able to offer you alternative financial services targeted towards you based on the information they have about your current financial products. I'm sure the financial companies offering their products on mint pay more than your average ad or click through.Most webapps don't have this sort of opportunity.
"Unlike most other webapps mint has a very lucrative revenue stream"
People keep saying this as if it were fact, but never provide evidence. From what I've seen, Mint has a great potential for revenue. I've not seen any public statements about their financials.
For a privately held entity, how exactly would you like a 3rd party to provide evidence? You have to either know someone on the inside or work by proxy. Assuming you don't know someone, your best proxy is the 10x valuation jump. The $145million valuation wouldn't come without proof of the revenue model to an unproven team.
They were executing revenue, and executing very well.
"For a privately held entity, how exactly would you like a 3rd party to provide evidence?"
If I knew how to answer that question, then we wouldn't be having this little debate. Fortunately, I'm not the one making claims about their profitability.
"They were executing revenue, and executing very well."
I'm sure the financial companies offering their products on mint pay more than your average ad or click through
You're right about that...these banks and credit card companies probably pay hundreds per lead, which is a drop in the bucket compared to how much they'll get over the life of the customer.
I would refer the high-speed-rail types to James Kunstler's brilliant post 'financial crisis called off' where he reminds us...
We blather about high speed rail, but you can't even get from Cleveland to Cincinnati on a regular train - and what's more amazing, nobody is really interested in making this happen. All we really care about is finding some miracle method to keep all the cars running.
part of me can't believe this was such an epiphany for this guy.
i've found the best way to screen candidates is to just ask them to sign up on project euler and work through at least three problems, in any language, and mail me the results. the quality of code i get so far is quite representative of the quality of the employee on the job.
yes, they could google the answers, but i disqualify the easiest-to-google questions (sum of multiples of 3 and 5 up to 1000, i'm lookin' at you). but honestly it doesn't matter - you can tell if they do because they can't explain it. and i can generally weed out anyone in the first place who would try to google the answer just from a skype conversation...
I have a similar completion rate and for a while I was really gung-ho about doing a Project Euler problem daily. Unfortunately, I discovered that there are sites which purport to have completed all 252(or maybe 253 now) and give out solutions.
That there are people who have cheated their way to high score, completely ruined the enjoyment of the site. It does seem a bit silly, that your own enjoyment of a problem is tarred by someone else's exploits. It shouldn't affect you, but it ultimately does, in some ways that someone else cheating in MMORPG affects one's own rat killing efforts.
That's really impressive. I'd interview anyone who showed such initiative. It's so common to interview "developers" who don't seem like they have any interest or passion about their work.
Hilarious. I was just scanning the headlines and thinking "WTF is up with all these Erlang articles?!" I'm trying to decide if posting this article is ironic or annoying.
Okay, fair enough, but I have a soft spot for Reader's Digest. When I was growing up in a small country a long time ago on a continent far away, Reader's Digest was a window into american life in easy-to-understand English. The coupons were for unimaginably weird stuff like Jello (I had no idea what it was, but the colours looked neat) and a zillion different kinds of painkiller (we just had aspirin). The stories were full of bizzare-to-me pre-occupations, like how to chose the right dentist.
And I still remember some of the funny stories. Here's one for any of you doing user support:
Two older folk are traveling and end up in this bar. It's packed, the music is so loud the poor waitress has trouble getting the orders right and the clientele is a bit rowdy. When the waitress comes to take their order, the old lady sympathetically says "My, you have a lot patience!" The waitress drops her voice and hisses "Shhh! We're supposed to call them customers!".
Our team is small and we typically have one big project at a time. Specialties are a) iOS, b) Rails, c) Erlang.
chad@inaka.net