TekMoi is right. I have an Alexa top 6k site which is vastly more complicated (media hosting, load balancing, multiple VMs, DDOS protection, transactional emails, automated backups) which costs $200 / month on AWS.
The fact that this person is spending nearly as much to support 50k users a day as I do to support more than 4 million cannot be hand-waived away by "people are so free to judge other's[sic] situations". The matter is worsened by the fact that the application is so simple that it doesn't even support user accounts. There is room for discussion here about efficiency in application architecture. More importantly, an article billing itself as "Costs of running a Python webapp for 55k monthly users" is silly because there is no way this is representative of anything. I'm afraid new hackers will be scared by the high costs listed here and be discouraged in their own efforts.
If you support 4 million monthly visitors on a media site, and have multiple EC2 instances running, I'd love to see a cost breakdown structure, because in my (obviously incomplete and possibly naive) calculations, the bandwidth alone would cost more than $200/mo.
CloudFlare covers the media bandwidth costs for a mere $20 / month. The uploading and media conversion is the difficult and costly (in terms of CPU) portion.
>CloudFlare covers the media bandwidth costs for a mere $20 / month.
Your media files must be extremely small.
I'm guessing they're less than 20MB on average, because a) CF hasn't shown you the door yet b) they don't even cache anything bigger than half a gigabyte.
Similar situation (sub $1k/month) to the setup im doing for a start up in indonesia (top 7k in alexa, top 150 in indonesia), except we have to pay extra for media because we need to process the logs (long tail pdf/data/docx hits, and we need to control the dns and how other domains route to use so we cant use cloudflare). And still plenty of room to cost optimize.
Another consultant that came in and try to do this, made our costs go up by 10x per month… so i'm not surprised when I see stuff like this here…
The knowledge of ones tools available at ones finger tips and the relative costs of such seem to make the difference for these things.
Let's leave this kind of snark out of this community. It's comments like this that slippery slope a community from helpful to harmful. You see it each time a reddit community gets too large.
No, what's harmful is "oh, just spend $50,000 on managed Kubernetes to run a Django web app". That costs real time and real money and makes young engineers think that a phpBB forum is impossible with a five-digit AWS bill.
Both you and TekMoi should probably value your own time higher. The cost savings are great, but once you spend 4 hours on configuring a database server, that'll probably be $200+ worth of your time and, thus, wipe out most of the savings.
It doesn’t cost anything to spend 4 hours of your time doing anything, so it doesn’t really wipe out any savings. Reducing a bill from $400 month to $200 is real money, not some theoretical time/value judgement.
And, as others have mentioned, there is enormous value in knowing how to operate on a fairly lean tech stack. It makes it so much simpler to scale effectively while keeping costs down.
This is only true, if you do not value your free time. In your example, you've spent 4h to save 200$, so your work was worth 50$/hour. A freelancer with a 100$/hour rate might do 2 hours of work instead, spend the money, and gain 2 hours of free time.
There are other factors of course, but in general, many people come up with a rate for their own free time (which is often higher than the actual rate they charge clients)
It’s not only true if you don’t your free time. It’s only true if you’re a freelancer that is turning down hours at a higher rate than what you’re saving.
Many people working on products, both on their own and within a company, aren’t turning down other profitable work to optimize existing solutions.
If I watch a two hour movie instead of spending two hours saving $100/month, it doesn’t matter how much I value my time, no one is paying me $100/hour to watch a movie.
There's a difference between the initial example and watching a movie. That's what I summed up with "there are other factors (than money)". If I'm only interested in the _outcome_ and I treat the way to achieve it as work, I definitely weight the time cost vs benefit (and I am not a freelancer). If it is something I enjoy doing (which may or may not be true for the initial example), I'll take this into account as well. Time is a limited resource, and I treat it as such.
Except that now you know how to configure a database server, and know how your database server is configured. So if you ever get a problem with the database in the future, you'll know how to solve it faster. Which will save you time and money in the future.
And you're less likely to make mistakes like upgrading your server instance to try and solve a problem that can't be solved like that.
And the more you do it, the cheaper and faster it gets. That knowledge and skill has value.
I value four hours of my time at considerably more than $200 and the reason I'm able to do that is because I know things like how to configure a database server.
But your argument is nonsensical because a one-time investment of time (much less than four hours for me, but I've been doing this for 20 years) can save you several hundred dollars a month. AWS AuroraDB, for instance, (which I also know how to configure, by the way) has much higher latency than a hand-rolled instance and will cause bottlenecks throughout your code in a DB-driven application. If I hadn't experienced the difference firsthand, or had failed to profile my app's performance adequately, I might assume I need to solve the problem by spinning up more ec2 instances to distribute the load. I've had the misfortune of working with a company that had exactly that problem and knowing how to spin up a new DB server saved the company thousands of dollars a month and took considerably less than four hours. Transferring a 3TB database to a new server without downtime did take considerably longer, however, but I was being paid hourly anyway, and it was still a worthwhile investment for the company which saved considerably more than my fee.
Any tradesperson should know their tools. A programmer is no different and if you don't know how to use your tools because your "value your own time higher" then thank you: you're the guy who ends up getting me called in to fix things at a much higher hourly fee.
the main issue I have with the DIY mentality is that, for me, it's endless, and even more so, it's an unpaved path, ad-hoc, with few integrated well defined paths. we've got lots of open source software, but the ops of being online is hard fought experience.
* `apt-get install postgres-server` would have worked fine for my needs on my VPS. oh but i need roles, so let's start an ansible playbook. maybe let's tweak some settings. fine, still all short, easy to do.
* ahhh i should probably have backups. how am i going to manage that storage, where is that going to live?
* then i introduce a new feature & my database is running slow. explain query helps, but i also could use some metrics for these boxes, so probably need to start thinking about prometheus & node-exporter, &c.
i am radically in favor of a) personally facing these challenges and b) open-sourcing the operational knowledge & tools for setting up AND OPERATING systems.
yet at the same time i also think spending $171/mo for a year is an exceedingly wonderful option to have on the table. running my own servers is, to me, a lifelong project, something i want to deeply invest in. there's plenty of ways to go about it that aren't so arduous (k8s+postgres-operator+rook+tbd monitoring+tbd directory-services), but that willingness to keep engaging, supporting, maintaining, scaling things can be a very serious concern that extends well past the time it takes to set a database up: it's an ongoing "giving a shit" burden even when (seemingly) working fine.
being willing and able to hack through is great, and i am all for the coallition of the willing who elect to march through, hopefully not getting bogged down along the way. but wow if you are trying to start a business, it sure is nice being able to pay someone to spin up, back up, monitor, scale some services for you.
i hope some day "we" are better at such things, systematically, i hope open source ops helps give us better paths to doing these kind of things easily, safely, observably, resilliently. we're not there yet. but wow, this challenge to me- how we move open source from an older "software" model to an online service model, that empowers people to set up online systems as easily as opening an editor, that's the challenge at the heart of open source today. it's one that needs a lot more effort, a lot more work, such that we have good ways to stand up & keep up a database server.
Being well versed in setting up your own database server bereft of cloud provider hand holding easily pays for initial time investment over time. I don't think most understand just how much a mastery of the basics is capable of generating in value when you're essentially vendor lock-in proof. There is so much blindness to voluntary hanging of one's arose out the window created by vendor overreliance.
It says in the article that the author gets 34k daily users, and 50k unique users/month. It would have been clearer if the author had talked about sessions (which are therefore > 1M/mo) for sure, but you're still making a very big (and invalid) assumption.
EDIT: Please disregard the above. I need an eye test, or maybe just to put my glasses on! Daily users are 3.4k (3400), not 34k. My apologies, I take it all back!
I'm such a klutz - sorry. This is what I get for reading HN when I'm still in bed and without putting my glasses on first. This is a terrible habit that I need to break.
Or 0 per minute and 25,000 per hour for two days a month. Traffic can be bursty; don't assume that X/month means they're getting exactly X/30 per day..
What are the right resources, you would suggest someone if he had to setup his servers properly. Will really appreciate if you can refer some books/videos/articles. Thanks.
If you're talking about serving many requests cost-effectively, then really the problem is not the servers (except over-provisioning, which is rampant. Learn to use tools like AWS' auto-scaling system instead) - it's the code.
If you understand the basics of algorithmic time complexity (that's your Big-O notation) and profiling your code then you're ahead of 98% of other developers in practice. I'm constantly amazed at how many developers think adding more libraries, newer frameworks, or more layers of tooling will magically speed up their code because "it's so fast". If you actually time things you'll find out doing it the "slow" way is frequently an order of magnitude faster.
The fact that this person is spending nearly as much to support 50k users a day as I do to support more than 4 million cannot be hand-waived away by "people are so free to judge other's[sic] situations". The matter is worsened by the fact that the application is so simple that it doesn't even support user accounts. There is room for discussion here about efficiency in application architecture. More importantly, an article billing itself as "Costs of running a Python webapp for 55k monthly users" is silly because there is no way this is representative of anything. I'm afraid new hackers will be scared by the high costs listed here and be discouraged in their own efforts.