It has often been alluded to that baseball or something like it started during the downtime between Civil War battles.. I doubt that the 6th, 9th, 20th, 54th or 55th Massachusetts regiments ever had downtime.
My home town attests to the earlier start of baseball. Please see the picture below. The actual building is still in use.
Try the Smithsonian for a complete early history.