We propose the first joint-task learning framework for brain
and vessel segmentation (JoB-VS) from Time-of-Flight Mag-
netic Resonance images. Unlike state-of-the-art vessel seg-
mentation methods, our approach avoids the pre-processing
step of implementing a model to extract the brain from the
volumetric input data. Skipping this additional step makes our
method an end-to-end vessel segmentation framework. JoB-
VS uses a lattice architecture that favors the segmentation of
structures of different scales (e.g., the brain and vessels). Its
segmentation head allows the simultaneous prediction of the
brain and vessel mask. Moreover, we generate data augmen-
tation with adversarial examples, which our results demon-
strate to enhance the performance. JoB-VS achieves 70.03%
mean AP and 69.09% F1-score in the OASIS-3 dataset and is
capable of generalizing the segmentation in the IXI dataset.
These results show the adequacy of JoB-VS for the challeng-
ing task of vessel segmentation in complete TOF-MRA im-
ages.