You don't
need ballast even with the mast up. I pull my son on a wakeboard, no ballast of course, with the mast up. It's more stable with ballast in, and the angle of vanishing stability is well beyond 90 degrees (laying on its side), and probably more so without the mast, but it's stable like a power boat without the ballast.
I keep the ballast in even without the mast (it's hanging off the ceiling in my garage) when using it in 'trawler mode', because it's less tippy, and I keep the rudders on so I can use them for slow speed maneuvers. I don't know how much better the fuel mileage would be without ballast, as it doesn't sit that much lower in the water with vs. without, and well below hull speed, it takes very little energy to push that boat.
Not having the mast to deal with does two things: It makes it much faster to launch and retrieve (good), and it makes it much more difficult to work on the deck, as there's nothing to hold on to other than the dodger (bad). It may be possible to rig a static line down the centerline of the boat, from where the mast roller would be, to the mast pin on the bow pulpit, and through any short mast you may add, to give something to hang on to. It would also be useful as a place to support a pair of bicycles, upright, like they do with the canal barges for rent here on the Erie Canal.
You also need to rig a masthead (steaming) light regardless. I have a long light-on-a-stick 2-mile all-round white light that does double-duty as masthead and stern light together for motoring with no mast (or mast down), and it serves as an anchor light regardless of mast status.