For about 10 years I had a consultancy that in the latter half focused on B2B. One of the biggest qualifier pitfalls was thinking we could replace Excel with anything else. There are a myriad of reasons this fails to work. I swear some of them are near voodoo.
Google Sheets is a fit in some situations but the real facts are that Excel is amazing and when someone has been running core biz logic on it for a long time; that operator is someone you need to get to know before feeling too certain.
There's money in replacing that person and fucking the whole thing up so they are stuck with a long tail of consultants. I never ran that sort of business. However on a couple painful occasions I overzealously pitched what python and psql could do to solve a problem and later had to eat crow.
I know what that voodoo may be. Of the many failed "replace Excel" projects I've seen in the investment banking \ hedge fund world over 20 years, the usual mistake is that the focus was on the specific Excel model itself and not the generic building capabilities that Excel provides. An Excel model, i.e., the specific map from inputs to outputs, is typically fairly easy to replace with another tool. But what you've done is replace just the pants, not the sewing machine. Excel is the pants AND the sewing machine. And that sewing machine part is very hard to replace. So you can end up with the same model in C++, the very same map from inputs to outputs, but the user's ability to make changes and understand the model is greatly diminished. So they say "no thanks" and stick with the Excel model.
You summed it up perfectly. Jobs I successfully finished after a few bites of tragically not understanding involved working closely with the Excel department and making sure they could function as usual but extracting data to go into the systems and providing a net gain for those people.
At the end of the day, the people that do the magic that keeps everyones job paying aren't always (or nearly never are) the people that make the contract hire decisions. If you can give those people a tool that lets them 2x their work and be excited about it there's a big net win. If you want them to move to a statically modeled web interface with dynamic tables and they lose all their ability to do everything they know in Excel, everyone loses.
Portability and timing is key if you have to take on that sort of gig. Team buy in might be worth more than that. Never take somebodies sewing machine if you still want that biz to make enough money to pay you.
Agree. If you're selective about which bits and pieces you move away from Excel, and not religious about getting rid of Excel entirely, you can often have a great result.
What trips people up is that Excel models can be deceptively inviting of replacement ideas. They may have many workflows within them that are obviously much easier to accomplish with a different tool. E.g., grouping and applying aggregate functions; filtering; joining; filling missing data, especially with constraints such as filling forward EPS estimates but only into the same fiscal quarter; etc. You could look at an ocean of INDEX(,MATCH(...)) functions in Excel and discover that the same manipulation in Pandas would require just two lines of code. But it is easy to overlook the importance of seemingly simple calculation flows, in the same model, that are uniquely suited for being expressed in Excel, the universal language for describing calculations.
For example, I have seen data analysts move from Excel to Python, replacing their models as they go -- after being wowed by Pandas' data manipulation capabilities -- and then get bogged down for months trying to recreate what Excel's =RTD("BLOOMBERG.RTD","",...) already did for them. And when the portfolio manager tells them, "this number looks wrong", which happens often, they spend half a day dumping data into Excel and building a sheet that's illustrative for the PM.
>Excel models can be deceptively inviting of replacement ideas.
In my mind that's actually what is the root at any talk of "Why isn't there an open source alternative for X amazing program?"
As a wiser person than me once said, "It's easy to imagine so I imagine it's easy." Deceptively simple operations in things that have a long legacy of supremacy can just be near impossible to clone even if you can churn out a POC that looks reasonable in 20 minutes. That trap is a big hill a lot of us die on. At one point in my life I swore off ever doing ETL, and now I do it for a living. Seems like that's actually the world now. Sheesh. With MS embracing Python for big data, maybe the next big thing is just what it's always been... Excel.
I worked six years in the IT department of a small domestic manufacturer, and deciding when a model had outgrown Excel and needed to be created as a tool (sometimes just a pretty interface for our ERP system) was one of our main tasks. I hate the hell that users can bring on a company when they misuse Excel but it will never go away.
We converted our excel wiz to do R. She will never return to excel for advanced stuff. The simple stuff could be done with lobte office (except for excel sheets that come from outside).
Google Sheets is a fit in some situations but the real facts are that Excel is amazing and when someone has been running core biz logic on it for a long time; that operator is someone you need to get to know before feeling too certain.
There's money in replacing that person and fucking the whole thing up so they are stuck with a long tail of consultants. I never ran that sort of business. However on a couple painful occasions I overzealously pitched what python and psql could do to solve a problem and later had to eat crow.