Is the axle worn? This could cause the problem.
Are you getting the axle nut tight?
If you have a steel splined bearing insert, then you will definitely where the axle splines.
Which is cheaper to replace, the axle or the alum splined insert?