Transcranial magnetic stimulation (TMS) is an effective treatment approach for mental disorders. Accurate and rapid prediction of TMS-evoked electric field (E-field) in brain tissue is important for accurate targeting and to understand the mechanism of treatment response. The standard method for E-field prediction is based on physical modeling which usually takes long computational time. In this work, we introduce a method based on deep neural networks (DNNs) for real-time E-field prediction. We show that the trained DNN can predict high-precision whole-brain E-field in 0.24 seconds. Moreover, diffusion-MRI based tissue conductivity tensor can improve the prediction accuracy of E-field.