From my understanding the seventh day of the week was the Sabbath in the OT. The 1st day of the week came to be the Christian Sabbath after the resurrection of Christ on the 1st day of the week.
I also was raised where Sunday was a day to go to Church and spend time with the family.
Back in the 1960's it was illegal for stores to be open on Sunday. If I recall correctly, the Blue Law came along in the late 60's and stores were allowed to open from 1 PM-6 PM on Sundays... Oh how times have changed.
I personally don't see anything wrong with dining out on Sundays. We do and I use the same argument as you.
The strick requirements concerning the Sabbath are in the OT. In the NT you read about Jesus teaching and healing on the Sabbath. He gave some examples that it is okay to do some work on the sabbath.