There may not be enough memory on the card to support the resolution you have set. Try rendering at a low resolution.
Here's the Next Limit statement on macOS:
Maxwell GPU rendering is not currently supported on macOS. For GPU rendering, Nvidia graphics cards with CUDA support are required, and Apple does not sell computers which include these at this time.
If you do have an older Mac model with an Nvidia card, we cannot guarantee you a great experience, because those cards are obsolete and have not been tested.
Should Apple include graphics cards appropriate for GPU rendering in the future, we will likely support these. We are also studying the possibility of supporting third party Thunderbolt GPU interfaces, which may provide a viable GPU option for our valued Mac customers.
Basically, powerful, contemporary, CUDA-enabled GPUs are required for this sort of work, and Apple is not producing machines that support such GPUs. However, I believe that Next Limit is researching a solution for users with old macs with Nvidia cards.
I think what we have to bear in mind is, that if Apple were producing computers suitable for this type of work, they would be absolutely supported out of the box. Many of us long time mac users are disappointed by Apple's lack of attention to the professional line of computers over the last several years. As a mac user myself, my plan is to keep using my mac for modeling and everyday tasks, and going to begin using windows boxes with powerful CPU and GPU on the network for my rendering.