Ion transport in materials is routinely probed through several experimental techniques, which introduce variability in reported ionic diffusivities and conductivities. The computational prediction of ionic diffusivities and conductivities helps in identifying good ionic conductors, and suitable solid electrolytes, thus establishing firm structure-property relationships. Machine-learned potentials are an attractive strategy extending the capabilities of accurate ab initio molecular dynamics to longer and larger simulations, enabling simulations of ion transport at low-temperature. However, being machine-learned potentials in their infancy, critical assessments of their predicting capabilities are seldom. Here, we identified the main factors controlling the quality of machine-learning potentials based on the moment tensor potential when applied to the properties of ion transport in solid electrolytes. Our results underline the importance of high-quality training sets in fitting moment tensor potentials, diverse training sets, and the importance of intrinsic defects which may occur in solid electrolytes. We demonstrate the limitations posed by short-time scale and high-temperature AIMD simulations to predict the room-temperature properties of materials.