A trench can act as a barrier to ground vibration and is a potential mitigation measure for low frequency vibration induced by surface railways. However, to be effective at very low frequencies the depth required becomes impractical. Nevertheless, for soil with a layered structure in the top few metres, if a trench can be arranged to cut through the upper, soft layer of soil, it can be effective in reducing the most important components of vibration from the trains. This study considers the possibility of using such a realistically feasible solution. Barriers containing a soft fill material are also considered. The study uses coupled finite element / boundary element models expressed in terms of the axial wavenumber. It is found to be important to include the track in the model as this determines how the load is distributed at the soil's surface which significantly affects the insertion loss of the barrier. Calculations are presented for a range of typical layered grounds in which the depth of the upper soil layer is varied. Variations in the width and depth of the trench or barrier are also considered. The results show that, in all ground conditions considered, the notional rectangular open trench performs best. The depth is the most important parameter whereas the width has only a small influence on its performance. More practical arrangements are also considered in which the sides of the trench are angled. Barriers consisting of a soft fill material are shown to be much less effective than an open trench but still have some potential benefit. It is found that the stiffness of the barrier material and not its impedance is the most important material parameter.