Gary Harkins scored the only goal of the game as poor finishing and Lactics keeper Mark Oxley denied Gillingham.
The Gills missed a glorious chance to take the lead when Myles Weston went clean through but could not beat Oxley.
Harkins then put Lee Johnson's Oldham in front with a powerful shot from close range after being found by David Worrall's deep cross.
Jon Stead hit the post for the hosts before Oxley saved from Adebayo Akinfenwa and Cody McDonald.
Oldham manager Lee Johnson told BBC Radio Manchester:
Post-match: Oldham manager Johnson
"It was a good win because sometimes you have to dog-out wins like that.
"Gillingham came here and made it a very difficult game for us. Mark Oxley pulled out a couple of saves, but then we had some really good chances.
"They are all tough in League One and I was just pleased we kept going until the end."
Gillingham manager Peter Taylor told BBC Radio Kent:
Post-match: Gillingham boss Taylor
"I can't believe we haven't won. We had a lot more chances than the opposition but their goalkeeper pulled off some wonderful saves.
"I don't think the players had a great deal of luck in front of goal.
"The second half in the week [against Sheffield United] was a disappointing second half and the players looked very tired. If two-and-a-half days later you're not going to take advantage of the squad, when are you going to take advantage of that?
"It probably means now that Peter Taylor has picked the wrong team because we lost the match but I don't think that was the case."
