Because back when CGI programs where the norm application servers weren't really a thing outside of maybe Java and similar ecosystem (by which I mean dotnet).
When you wrote a web application before the ~2010s you wouldn't start of with a library with webserver building blocks to which to attach routing + behaviour. It sounded crazy at that time, especially when you where building websites with a scripting language like PHP/Perl/Python/Ruby. It sorta became the norm with the rise in popularity of event driven IO, through projects like NodeJS and the rising popularity of Go.
Even today, at least in the PHP world, we still mostly use the CGI based approach to running code (though it's through a more modern interface such as "fast cgi"), with an inbetween service that handles process (fpm - fastcgi process manager). I'm sure other dynamic languages have something similar, though there is a large wave of people pushing for more application server style deployments in those ecosystems as well (which is more feasible today).
When you wrote a web application before the ~2010s you wouldn't start of with a library with webserver building blocks to which to attach routing + behaviour. It sounded crazy at that time, especially when you where building websites with a scripting language like PHP/Perl/Python/Ruby. It sorta became the norm with the rise in popularity of event driven IO, through projects like NodeJS and the rising popularity of Go.
Even today, at least in the PHP world, we still mostly use the CGI based approach to running code (though it's through a more modern interface such as "fast cgi"), with an inbetween service that handles process (fpm - fastcgi process manager). I'm sure other dynamic languages have something similar, though there is a large wave of people pushing for more application server style deployments in those ecosystems as well (which is more feasible today).