Maybe a dumb question, but could one use bricks of depleted fission fuel as the neutron shield in a fusion reactor, and swap them back into a fission reactor every time they become fuel-grade again?

Sorry, maybe I was unclear. I wasn't asking about using the fusion reactor just to produce fission fuel. I was asking "if the fusion reactor produces a bunch of extra neutrons as a side effect, is this a way to use them to produce more energy later instead of throwing away that energy, or letting the neutrons actively damage the fusion reactor?"

i.e. one needs to prevent the neutrons from escaping or degrading the material the reactor is made out of, so instead of just adding a throwaway shielding layer, would it work to use something that is not only super dense, but is currently a waste product[1], and also becomes more valuable in the process of being used as the shield? Seems like it would be vaguely equivalent to a photoelectric collector for neutrons, but I am not a physicist or reactor engineer.

[1] I do think it's a shame that we stopped building breeder reactors, but that's a separate discussion.

Breeder reactors are not used because refining the bred fuel costs more than processing mined uranium.

The overarching requirement on fusion neutron absorption material, besides delivery of heat to process steam, is that it needs to produce more tritium to burn. You don't get that if the neutrons are absorbed in something other than lithium.