Metasurfaces are composed of a two-dimensional array of carefully engineered subwavelength structures. They provide a novel compact alternative to conventional voluminous optical components. However, their design involves a time-consuming hit and trial procedure, requiring many iterative electromagnetic simulations through expensive commercial solvers. To overcome this non-practical design strategy, recently, various deep-learning-based fast and low computational cost networks have been proposed to design and optimize individual meta-atoms and complete metasurfaces. Most of them focus on optimizing the amplitude response of nanostructures, whereas mapping the phase response is a much more challenging problem that needs to be addressed. Since the metaatom's optical response is entirely reliant on and vulnerable to its geometrical structure, underlying material, and operating wavelength, changing any of these parameters changes the entire physics of the problem in hand. Here, we propose novel deep-learning-based generalized forward and inverse design approaches to optimize all-dielectric transmissive metasurfaces. The proposed forward predicting neural networks take all the geometrical parameters and the physical properties of the bar-shaped dielectric nano-resonators as the input and predict the cross-polarized transmission amplitude and modulated phase at eight distinct rotation angles of the nano-bar. These networks are generalized to predict the electromagnetic (EM) response of different dielectric materials at different operating wavelengths. An inverse design neural network is also proposed that takes the target transmission amplitude and phase at eight discrete orientation angles of the nano-bar as the primary input. The underlying physics of the problem is also incorporated by feeding the intrinsic material properties and the operating wavelength of the nano-bar as a second input to the inverse neural network. It predicts the optimum set of geometrical parameters to achieve maximum cross-polarized transmission and complete Pancharatnam-Berry (PB) phase coverage from 0 to 2π. The average test data mean square error (MSE) achieved for the forward predicting neural network is 1.8 × 10−3 and that of the inverse design neural network is 2.8 × 10−1. The average MSEs for different material's test samples are demonstrated to validate the generalizability of the proposed models in terms of seen and unseen materials. A comparative analysis of the proposed approach with conventional EM software optimization tools is performed to prove that the proposed inverse design works much faster than the conventional methods, also it can handle a comparatively larger range of parameters and predicts the results in a single run with high accuracy.