Yes you are right it's applied to the parameters, but other models (like ngcm) applied it to the inputs. IMO it shouldn't make a huge difference main point is you max differences between models.
I wonder, what LeCun wants to do is more fundamental research, i.e. where the timeline to being useful is much longer, maybe 5-10 years at least, and also much more uncertain.
How does this fit together with a startup? Would investors happily invest into this knowing not to expect anything in return for at least the next 5-10 years?
That's a quite different thing, OpenAI has billions of USD/year cash flow, and when you have that there's many many potential way to achieve profitability on different time horizons. It's not a situation of chance but a situation of choice.
Anyway, how much that matters for an investor is hard to form a clear answer to - investors are after all not directly looking for profitability as such, but for valuation growth. The two are linked but not the same -- any investor in OpenAI today probably also places themselves into a game of chance, betting on OpenAI making more breakthroughs and increasing the cash flow even more -- not just becoming profitable at the same rate of cash flow. So there's still some of the same risk baked into this investment.
But with a new startup like LeCun's is going to be, it's 100% on the risk side and 0% on the optionality side. The path to profitability for a startup would be something like 1) a breakthrough is made 2) that breakthrough is utilized in a way that generates cash flow 3) the company becomes profitable (and at this point hopefully the valuation is good.)
There's a lot of things that can go wrong at every step here (aside from the obvious), including e.g. making a breakthrough that doesn't represent a defensible mote for your startup, failing to build the structure of the business necessary to generate cashflow, ... OpenAI et al already have a lot of that behind them, and while that doesn't mean that they don't face upcoming risks and challenges, the huge amount of cashflow they have available helps them overcome these issues far more easily than a startup, which will stop solving problems if you stop feeding money into it.
> That's a quite different thing, OpenAI has billions of USD/year cash flow, and when you have that there's many many potential way to achieve profitability on different time horizons. It's not a situation of chance but a situation of choice.
Talk is cheap. Until they're actually cash flow positive, I'll believe it when I see it
I was looking for various options to archive my data (photos, documents, code), and have looked at Perkeep since a while, but then started using Git-Annex.
However, I regret this decision. Git-Annex is not usable anymore on my data because the amount of files has grown so much (millions) and Git-Annex is just too slow (it takes minutes up to even hours for some Git operation, and the FS is decently fast). I assume I would not have had those problems with Perkeep.
It reminds me also of the 24 FPS discussion, which is still the standard as far as I know for cinema, even though 48 or 60 FPS are pretty standard for series, The 24 FPS give it a more cinematic feeling.
To add, when it comes to video games sometimes people go "baww but 24 fps is enough for film". However, pause a film and you'll see a lot of smearing, not unlike cartoon animation frames I suppose, but in a video game every frame is discrete so low framerate becomes visually a lot more apparent.
I think it was The Hobbit that had a 60 fps version, and people just... weren't having it. It's technologically superior I'm sure (as would higher frame rates be), but it just becomes too "real" then. IIRC they also had to really update their make-up game because on higher frame rates and / or resolutions people can see everything.
Mind you, watching older TV shows nowadays is interesting; I think they were able to scan the original film for e.g. the X Files and make a HD or 4K version of it, and unlike back in the day, nowadays you can make out all the fine details of the actor's skin and the like. Part high definition, part watching it on a 4K screen instead of a CRT TV.
Regarding your first paragraph, the other thing worth mentioning is that in addition to smear/motion blur in film, it's also a passive medium. Video games are interactive. You're responding to and making decisions based on what you're seeing.
In a FPS, trying to track movement at only 24 fps is pretty much impossible unless your target's movement is entirely predictable.
In a flight simulator, trying to land a plane in gusty weather conditions is a lot harder with only 24 fps.
Lower framerates don't just make motion choppy, it increases latency. At 24 fps, any change in movement could be up to 42 ms behind. At 120 fps, that's down to 8.3 ms. And those numbers assume that you can notice the difference in only a single frame.
I'm convinced that people claiming 24 fps is fine for games just because it's fine for film don't actually play games. At least, nothing that requires quick reaction times.
I have asked about using RPython as a generic standalone language before. I think the official statement is that is was never intended to become one, and it's really a very minimal subset of Python (so basically no existing Python code will run, it would require heavy refactoring or complete rewrite), and it's only specifically those features that they currently need, and it might also be a moving target, and they don't want to give certain guarantees on stability of the language etc.
Once you consider that you anyway need to write very different kind of code for RPython, then maybe just using Nim or some other language is a better idea?
A language can be suitable for writing a compiler, but if there is another language that's 10x faster that's also suitable, then you're losing out on a lot of compilation speed for no reason.
Dog-fooding a language by writing a compiler in it can lead to the designers adding language features to make compiler development easier, even if they detract from the design of the language for the 99% of users who aren't writing a compiler.
QtCreator was a bit like the lightweight version of KDevelop for me. I didn't really needed any of the Qt features, just the C++ editor. And the C++ support was really good.
In my case, it was easier to identify: I had another implementation of my loss function before that did not use masked_select. But then I thought I can be clever and use masked_select to take out the non-masked frames and calculate the loss only on those. But it wasn't working. Also, it only happened for some models, not for all. It turns out, it was always happening when the data coming out of the model was non-contiguous.
I think the bugs with non-contiguous data are not so uncommon. I wonder how much of that we still have.
