It’s a fork bomb. It creates and calls a function called : that calls itself and pipes the output to itself in the background.
Properly indented and commented, it would look something like:
:() # create a function called ":" with no arguments
{ # start the function
: | : & # call ":", pipe the output to another invocation of ":", and run in the background
}; # end the function
:
P.S. Don't try this at home
It is a common misconception that today people can just "shoot a vm / container in the head" to solve all problems, but if your problem is buggy code then the new VM / container will crash the moment the code executes. Diagnosing and resolving quickly gets harder without system limits in place.
[1] - https://en.wikipedia.org/wiki/Fork_bomb
[2] - https://serverfault.com/search?q=cgroup+limits