You can, but it's not a good idea, generally. It's always safer to use a separate, portable tank for your kicker. That way, if you have a fuel contamination problem in your onboard tank, you can still run the kicker. It's a more common reason for an engine quitting on the water than you'd think.
Your 4hp will run a long, long time, even on a 3.5 gal tank.