It actually happened in "The Mighty Thor" #140 from 1967.
Comics were a lot different in 1967; fantastic, supernatural events were routine plot devices, but even then Marvel was the more 'scientific' of the two large superhero comics companies. Marvel loved genetic mutations and scientists were often heroes or villains but decades ago the resolution of the plot was going to be fast and likely something of a letdown.
Halfway through I put the book down and wondered 'How would I defeat someone who got stronger due to force?
First, we have to create some parameters. In comic book science, as in any fictional narrative, you have to limit your 'magic' in order to maintain suspension of disbelief by the audience. So while the audience can accept that Thor will be super-strong, that 'God' business is dismissed as the beliefs of ancient people who didn't understand science. We can see Asgard with the Hubble, in modern day comics lore.(1)
At times like this I invoke Bertrand Russell's proposition that once a contradiction is allowed into a closed system anything can be proven. It is said that during a talk on this matter, someone in the audience yelled out a challenge: "If 2 plus 2 equals 5, prove that I am the Pope."
Russell replied, "If 2 plus 2 is 5, then 4 is 5; if 4 is 5, then (subtracting three from each side) 1 is 2; you and the Pope are two, therefore you and the Pope are one."
You see how it works - and how it doesn't work. If 2 plus 2 equals 5, okay, but it 2 plus 2 equals everything, then it equals nothing. You also don't play baseball with two balls at the same time so we won't introduce two contradictions, like that the bad guy is all an illusion or something. Instead, let's tackle what would happen if Thor kept clobbering him and he kept growing.
Stan Lee and Jack Kirby conflate size with strength in this case. The larger the villain gets, the more damage he can do. But what they didn't know is that the 'larger' you get, the less strong you are going to be, relatively speaking, including in your ability to support your own weight. The reason is that while the strength of a muscle is proportional to the surface area of its cross section (two-dimensional, proportional to the square of its length), the weight of that muscle is determined by its volume (three-dimensional, proportional to the cube of its length.)
Strength scales by the cross-sectional area and while exercise makes a muscle bigger, it does not make it longer. So every time this giant increases his size by a factor of 3, he only increases his strength by a factor of 2.
That's how Thor can win and that's why giant mutant ants - sorry, 1950s science fiction - wouldn't even be able to move, even though they are incredibly strong at their tiny size.
This Asian weaver ant is lifting 100X its body weight - upside down and hanging from a smooth surface. Thor is lucky he is not fighting one of these critters. Credit and link: Popular Science
Adding a factor of 3 for weight for every factor of 2 in strength means diminishing returns kick in rather quickly. Rather than stop hitting the giant because it is making him bigger and stronger, the smart approach is to hit him even more - because he contains some sort of transducer than converts the force of the hammer strikes to mass, he can't control it, and that will be his undoing.
How many more hits would Thor need to deliver in order to make his size increase so much he collapses? That depends partly on the mass and velocity of the hammer doing the delivery. Dr. Neil deGrasse Tyson once said that, based on his understanding of its material, Thor's hammer must weigh as much as a herd of 300-billion elephants but we know that isn't so, because it was not forged from the heart of a dying neutron star, it was forged in it. The actual weight, according to specs, is 42.3 pounds:
Credit and link: Urbandud via Matt Shipman at NC State.
Let's do some comic book math. We know kinetic energy is ½mv2. The bad guy looks like he is flying about 30 feet so for the sake of the international audience, let's say 10 meters - Americans are used to working with both. A 42 pound hammer is also 19 kilograms.
The creature was clearly already big by that point, around 3000 Kg if he is primarily an Earthly metal and about the same density as a truck (so, a lot like a human, but structured of metal with lots of other stuff around the skeleton.)
But Thor is really strong. If he can take out Hercules, he can take out a truck. Humans are 'strong' anyway, a good boxer can generate up to 5,000 newtons of force while a good martial arts kick will be 9,000 - but Thor is rated in something Marvel calls Strength Class Incalculable, able to lift over 100 tons. That sounds impressive but going by the Marvel Wiki half of their universe can do that.(2) Strength is not the only factor in force, of course, but it sure doesn't hurt.
The hammer is basically irrelevant at that point, with or without it Thor is doing some terrific damage if a 230 lb. or so man can lift 100 tons. Making a 3000 Kg bad guy go 10 meters is trivial, a scant 400 horsepower engine could pull it off. 3000 Kg X 9.8 m/s2 X 10 meters in 1 second = 294 KW, or 400 HP.
The question becomes how many times does our hero have to knock him around before his increased size, due to the force Thor is generating against him that is being converted to mass, overwhelms that increased size. So if the big guy was 3,000 Kg and his relative strength was twice that (he is able to hold himself up, so it can't be less, but the worry was not that he would overwhelm Thor physically, rather that he would grow so big he would destroy the planet), the next hit would make his relative strength 36,000,000 but his mass is already 27,000,000,000. He is being crushed under his own weight. One more hit beyond that and he is a puddle of futuristic goop in the crater he created with his weight on the hit before.
So Thor needn't have worried about lending it his thews - physics was going to do the real work for him. Come on Asgardian, break some stuff once in a while.
That's my solution, anyway - and if I ever encounter a giant ant, just give me a chalkboard and I will prove to him that he couldn't have gotten that big. Then I'll run like crazy.
So how did they resolve it in the book? Kang, the conqueror from the future, had created it as a weapon but hidden it in the past and he shows up and makes the big robot shrink back to doll size again so he can hide it from his enemies. So the plot was resolved quickly but even then they threw in some science, to keep Kang from using it against them later:
Thor #140 in 1967. Credit: Marvel Comics Group
Universal Infinity Vortex? How has that not been made into a TIME story about climate change or at least gotten a Weather Channel storm name in all these decades?
Anyway, a hammer that can create a faster-than-light continuum and thus a cataclysmic displacement creates a lot more natural laws problems than it solves. Occam's Razor says that clearly the better solution would be to have had Professor X zap him into unconsciousness telepathically.
(1) And sometimes it just looks like we are seeing Asgard.
This is 'Mystic Mountain', 7500 light-years away in the southern constellation of Carina. Credit: NASA, ESA, M. Livio and the Hubble 20th Anniversary Team (STScI)
(2) Ant Man? Seriously? They need to read this article about the ant thing.