Can offense hit the ball out of bounds on a fumble?

me and my friends were playing football and the quarterback fumbled on a 3rd down. he kept hitting the ball backwards and to the side(not picking it up) and blocking me (the defensiv line) until the ball was out of bounds. is this legal or is it a foul. because if not a foul then they would just get the ball back at the old line of scrimmage and it would be fourth down. we played this way and he scored a touchdown on 4th down. it made me mad. so i really want to know if this is legal.