Most of the advancements with deep learning have come from mapping the reconstructed MRl images to outcomes (e.g. tumor segmentation, survival rate, pathology risk map). In this work, we present methods to arrive at critical medical imaging tasks like segmentation, classification directly from raw k-space data without image reconstruction. We specifically demonstrate that from k-space MRI data, we can perform hippocampus segmentation as well as detection of motion affected scans with similar performance to that obtained from imaging data. We also demonstrate that such an approach is more resilient to localized artifacts (e.g signal loss in hippocampus due to metal).