the problem with rarified air is that only Ferrari can race Ferraris. Given the low volume, high prices, and resultant engineering issues (cough cough overheating) that wouldn't exist in cars more on the mass market, requiring teams of engineers to support.
The average car enthusiast for track days will be seen running Mustangs, Corvette/Camaros, BMW M2/4s, Toyota GR 86s/Supras/Corollas, Honda S2000s, Nissan Z's, etc. But Ferrari is essentially irrelevant.
Sadly, I think Porsche is going that way; before, one could also run with the above gaggle with a decent 911 or Cayman. Now that the entry level is climbing well above the $100k mark new, and what was supposed to be their track car (GT3) is approaching Ferrari prices, they are becoming less and less relevant. And they might not even survive the China situation...
I'm not too familiar with the history of a few that you've mentioned, but the Mustang (excluding the GTD), Supra, Carolla, S2000, Z never had track focused cars. For the time-poor and money-rich racing enthusiast, lacking a turn key package isn't really an option.
Your local track days where you see the likes that you mentioned - you'll also likely get a few Porches etc too. But there's absolutely a culture of racing higher end cars, the GT3's, Ferraris (488 Pista, 360 Challenge Stradale, starting to see XX's around too), the Valkyrie and Valiant etc. There's also those that grab older Cup cars and track them to the track. There's absolutely a market for turn key, track focused, road legal Ferrari's.
Ferrari is a very rare thing in modern motorsport. A team that races to fund racing.