My ex worked at a startup where she was hired as a the second or third data scientist. Their entire Posgress database dump was 20 MB. And they had three people working full time on analyzing ... that 20 MB.

I call it artisanal data science *chefs kiss*.

Data lake? More like data pond

this was me in 2017 lol

It’s a great skill to walk in to a job and say “hey I’m the expert, that’s not a reasonable proposal, here’s the problem we can solve and here’s what we’ll do”. Much more value to the company, but hard to do.

It’s also not rewarded in modern companies. People are rewarded more for worthless garbage produced than worthless garbage avoided. You don’t have much to show for yourself when you talk a company down from making a plunge into a foolhardy, doomed initiative. Pretty soon the bean-counters might wonder why you are being paid when you don’t have as much to show for your work as others. See Elon’s “lines of code” decision making.

Yeah I feel a lot of companies could do with running their problems past a consultant first.

Also, w.r.t hiring in cases like these, I think often the experienced candidates can smell that this won't be a good gig so don't apply, while the less experienced (or desperate) ones apply. This means the workers get stuck with an intractable problem, and the company gets stuck with workers who are too inexperienced to know better.

> even the need, for ____ ________ yet try and hire them anyway.

I think that vast majority of human organizational structures, individuals to large corporations and countries have no clue what they are doing. The most successful apply science and just barely keep their head below the surface by avoiding utter failure. Most people would describe the Olympics as a competition to find out who is the best amateur in a given sport. No, the Olympics is a competition to see who can make the least number of mistakes.

If you are going to make bold dumb moves, you need a whole lot of margin.