I think you may have cracked latent space reasoning. I've had a hunch that something like this would work, but couldn't figure out how the training would back propagate. But you've shown that you just need to duplicate existing layers.
Have you tried a simple inline loop over the duplicated layers? Would be interesting to see performance. Also, would be interesting to compare with a MOE model. See if these layers are acting like different agreeing "experts" or if there is reasoning happening in the latent space.
Yes, I've tried duplicating indvidual layers, but its not useful.
I think this hasn't been tried before because it's totally unintuitive that feeding the output from later layers into previous ones would actually do anything. And in fact, it usually is detrimental. I guess it takes really bored hobbyists with too much compute to check this stuff.
I have done some interesting work on applying multiple layer duplications in different regions of the model too, going so far as to train a meta-model (actually just XGBoost) to predict the merges. Seems to work, buts thats a whole other blog post.
This works with MoE, and yes, I would be interested in looking into this in more detail. But my wife might disagree with this time sink...
This post has been sitting in my writing drafts for about a year, but i’ve only just gotten around to publishing it. Some of the information may be a bit out of date, but I’ve tried to keep the principles timeless. Hope you find it helpful
One gem if you're interested in semiconductors is the Feynman lecture "There's plenty of room at the bottom." He basically laid out the case for the modern nanotechnology age in 1959
I agree with this entire article, except that it's not just a right-wing phenomenon. Both sides distort the truth for financial gain and from a quantitative POV it's not a false equivalence.
At least in the U.S., from my observations, it's mainly a right-wing phenomenon and borne from the same culture that brought us the likes of Glenn Beck, Rush Limbaugh and Hal Turner. As a teenager, my friends and I would laugh at this stuff and assume by the time we were having kids, Fox News and similar right-wing, vaguely racist, puritanical media would be diminished to a point where it was hardly noticeable. We were clearly very wrong.
To quote you from an earlier comment of yours: "This is exactly the sensational take (devoid of nuance and information) that we should collectively push back against."
The left has been traditionally anti-capitalist and in favor of improving rights and living conditions. Who on the left is gaining financially from distorting the truth to the level of someone like Larry Ellison, Elon Musk, Peter Thiel, the Koch Brothers, or Jeff Bezos?
In the favor of improving living conditions? If and only if it is through their ideology. If it is through something which goes against their ideology the goalposts move at the speed of light and out comes some rationalization like "it wasn't really important, what we need is community".
When communism had claims to being more productive growth was the most important thing in the world and why we should adopt their ideology. Now look at the 'degrowth' people what a coincidence and they are literally arguing for worsening living conditions....
That's because Marx was a productivist, and communism was strictly productivist until a decade ago (and still is mainly productivist nowadays).
In a lot of country political ecology used to be liberal/capitalist (save a few radical feminist like D'Eaubonne who linked environment with feminism, but it's less than a minority). Basically Blair's 'third way' but with less nuclear (for some reason, although I think this position is loosing ground in ecologist), and more electric cars.
The degrowth movement is an offshot of that ideology. Degrowth is to political ecologism what anarchism is to communism, based a very Idealist and hopeful view of humanity.
Communists and ecologists are broadly on the 'left', but rarely allied until maybe a decade ago, and again, on minor things (Communists love nuclear,as it is typically something you don't want a capitalist with 'limited liability' to take care of), and while degrowth might be close to anarchists in some way, it is very, very dishonest to put them in the same basket.
True, but rarely you see someone grifting to the left. Each time someone get caught putting his hands, or worse, where he shouldn't, he tends to go rightward, and fast. Same thing when they are caught doing MLMs, always to the right. Even with shitcoiners, the right-wing pivot is fast once your 'racism is bad, but this antiracist coin' is done. I honestly don't know why.
Probably you have either more money, or more forgiving/powerful people on the right? Some grift to the center, but it's very rare, when libs are as powerful and rich as conservative, so I've no idea.
Sorry, i wasn't clear, my bad. You have grifters everywhere, especially in the fringe as when your audience is small, grifting is the best way to make any real amount of money. But when a grifter is caught doing something the public consider bad (SA, but sometime just being caught red-handed stealing money), he either disappear, or grift to the right. No grifter go further left is my point.
I think the internationally known examples are the Tate brothers or the Paul brothers, but among the scammers who got pardonned by trump you will mostly find people who grifted to the right. In my country it is also quite clear. Doctors caught doing weird chamanism and not helping patient? Suddenly it's "cancel culture!!1!". Your "quantum energy" bullshit is caught and proved to be a weird MLM? Cancel culture! Every fucking time.
It happened with people who used to grift for kombaya/hippies with weird chamanism and yoga. Suddenly, 911 was an inside job and Donald Trump is our savior, Q is his prophete. I shit you not, my mother Yoga ayurvedic group went from hippies that use to get scammed into buying essential oils to "open the Chakra" to weird antisemites who purchased trumpm/melania coin in less than 3 years. And we're not even in the US :/
Ah I think i understand. To paraphrase, you're saying that people that are caught tend to take refuge by more rightward grifting.
I cannot speak to your specific experience and i don't necessarily disagree. However, I do think there a potential selection bias here because by our media environment. Agree or not, the media believes that Trump is the most horrible person in the world at the moment. So there is a propensity to attack anyone that is considered allied to the right, so rightward grifters will get more aggressively exposed.
Yes, exactly, I'm not a native speaker and I have trouble explaining nuances on stuff I've thought about in my mother tongue first.
Agree with the selection bias, that one explanation. A second is that a lot of grifters who were caught now want to get pardoned, so maybe it influence their politics. And the third one is that the society as a whole shifted right, so grifters shifted right. When the pendulum will swing back, maybe grifters will also be shifting back (it doesn't match with Russell Brand/Andrew Tate timing though, but it match quite well with the podcast bro crowd and the Paul brothers).
I also think the age and money factor is also a thing (conservative have more disposable income on average, are older on average, so easier and richer targets). When you get caught grifting your audience will decrease. Concentrating on the richest/oldest is just good business.
New datacenter projects are usually closed loop now.
From your first citation:
> Closed-loop cooling systems enable the reuse of both recycled wastewater and freshwater, allowing water supplies to be used multiple times. A cooling tower can use external air to cool the heated water, allowing it to return to its original temperature. These systems can reduce freshwater use by up to 70%.
Citation please, I don’t buy it. Evaporative cooling towers almost double the efficiency of heat rejection vs a closed loop system. I don’t see any data center operator giving up those operating cost efficiency gains just to save some water, but I could be wrong.
Yeah, I still don’t buy it. No data center operator is going to use almost twice as much electricity to operate their cooling system if they could install evaporative cooling towers and almost double the efficiency of heat rejection. You can remove 7W of heat for 1W of electricity with a cooling tower, it’s 4W for 1W with a closed loop system. I could see them doing it in a humid area where you can barely get any cooling from evaporation due to the air being satured, but that only covers the Deep South and Florida in the US.
I don’t care if some company operating data centers claim that they’re aiming for closed loop cooling systems when the economics favor open loop cooling. Show me the incentive and I’ll show you the outcome, the incentive to increase profit by using cooling towers will beat out the abstract good feeling you get from not using evaporative cooling towers and paying almost twice as much to reject heat from your data center which you only operate to make a profit.
I think this is the difference between research and industry. Industry should try to grind out obvious improvements through brute force iteration. I really wish the culture of academia was more of an aim towards moonshots (high risk, high reward).
Fun quotes from the paper
> I. Institutions Are Society’s
Superheroes: Institutions are essential for structuring complex human interactions and enabling stable, just, and prosperous societies.
> Institutions like higher education, medecine, and law inform the stable and predictable patterns of behavior within organizations such as schools, hospitals, and courts., respectively,, thereby reducing chaos and friction.
>Similarly, journalism, as an institution, commits to truth-telling as a common
purpose and performs that function through fact-checking and other
organizational roles and structures. Newspapers or other media sources lose
legitimacy when they fail to publish errata or publish lies as news.
> Attending physicians and hospital administrators may each individually possess specific knowledge, but it is together, within the practices and purposive work of hospitals, and through delegation, deference, and persistent reinforcement of evaluative practices, that they accomplish the purpose of the institution
> The second affordance of institutional doom is that AI systems short-
circuit institutional decisionmaking by delegating important moral choices to AI
developers.
>Admittedly, our institutions have been fragile and ineffective for some time.36 Slow and expensive institutions frustrate people and weaken societal trust and legitimacy.37 Fixes are necessary.
> The so-called U.S. “Department of Government Efficiency” (“DOGE”)
will be a textbook example of how the affordances of AI lead to institutional
rot. DOGE used AI to surveil government employees, target immigrants, and
combine and analyze federal data that had, up to that point, intentionally been
kept separate for privacy and due process purposes.
This is exactly the sensational take (devoid of nuance and information) that we should collectively push back against. You should read the actual guardian article that Vice links to and investigate the actual reality.
If you actually look at the data, the vast majority of expenditures in medicine are the last few months of life. Paying ~500K to 1M to extend life ~6 months so you can spend the time hooked up to tubes and passing in and out of consciousness to deal with the pain is NOT HUMANE.
Have you tried a simple inline loop over the duplicated layers? Would be interesting to see performance. Also, would be interesting to compare with a MOE model. See if these layers are acting like different agreeing "experts" or if there is reasoning happening in the latent space.
reply