Unfortunately, you do owe him for all the hours he worked. It may not seem fair, but it's the law under the Fair Labor Standards Act (FLSA).

Employees are entitled to be paid for all hours worked at the rate agreed upon. If you fail to pay him, he can obtain assistance from the Department of Labor to enforce his claim.

An argument might be made if you had a written contract with him to complete a specific project or task that he failed to do; but even in that instance, he might make a good case that he should be paid for the effort he expended in your employ.

For seasonal workers, sometimes it's a good idea to offer a lower wage with the commitment of a sizable bonus upon them completing the season.  That "carrot" might help you to keep the workers you need for the time you need to have them on board.