Infinity is not a number but transfinite numbers [1] like omega [2] or aleph-null [3] are. They are just as number as 0, 1, -5, 6/7, pi, i, or quaternion numbers. What is a "number" is not well-defined, any mathematical algebraic object can be called a number with some justification. Mathematicians just don't do it anymore, they just call it "X Algebra" instead of "X Numbers" but there is no fundamental distinction. From this perspective, arithmetic with infinity is not only possible but also extremely useful for Programming Language Theory, e.g. for Dependent Type Theory.
[1] https://en.wikipedia.org/wiki/Transfinite_number
[2] https://en.wikipedia.org/wiki/Ordinal_number#Ordinals_extend...
[3] https://en.wikipedia.org/wiki/Aleph_number#Aleph-null