As part of the development of many robotic systems for the forestry sector, forest scene understanding requires the use of computer vision algorithms. However, this dense and unstructured environment is complex and puts conventional detection approaches to the test. In the case of tree instance segmentation, the presence of closely spaced or even intertwined trees, their highly variable shapes, and complex masks due to their branches and leaves are just some of the challenges to be overcome. For this, specific learning of tree boundaries is required to better distinguish one from another. In this paper, we propose ConvexMask, a convolutional neural network for real-time instance segmentation. ConvexMask opts for a label representation approach with a convex exterior polygon, defined by tree extremities, and a binary mask to handle the detail and occlusions that the label may contain. Experiments conducted on the SynthTree43k dataset show that ConvexMask distinguishes tree extremities better than state-of-the-art networks, resulting in better-quality masks. The code is available at https://github.com/rcondat/convexmask