Referring more to smaller places like my own - few hundred employees with ~20 person IT team (~10 developers).
I read enough about testing that it seems industry standard. But whenever I talk to coworkers and my EM, it’s generally, “That would be nice, but it’s not practical for our size and the business would allow us to slow down for that.” We have ~5 manual testers, so things aren’t considered “untested”, but issues still frequently slip through. It’s insurance software so at least bugs aren’t killing people, but our quality still freaks me out a bit.
I try to write automated tests for my own code, since it seems valuable, but I avoid it whenever it’s not straightforward. I’ve read books on testing, but they generally feel like either toy examples or far more effort than my company would be willing to spend. Over time I’m wondering if I’m just overly idealistic, and automated testing is more of a FAANG / bigger company thing.
Yeah, but it isn’t usually very difficult to write a test correctly, unit tests especially.
If you can’t write a test to validate the behaviour that you know your application needs to exhibit, then you probably can’t write the code to create that behaviour in the first place. Or, in a less binary sense, if you would write a test which isn’t “right”, you’re probably just as likely to have written code that isn’t “right”.
At least in the case with the test, you write the test and the code, and when the test fails (or, doesn’t fail when it should) you’re tipped off to something being funky.
I’m sure you could end up writing a test that’s bad in just the right way to end up doing more harm than good, but I do think that’s the exception(heh).
That’s exactly why I’ve asked. That is where I’ve gone wrong with TDD in the past, especially where any sort of math is involved due to being absolutely horrible at it (and I do game dev these days!). I can problem solve and write the code, I just can’t manually proof the math without external help and have spent countless hours looking for where my issue was due to being 100% certain that the formula or algorithm was correct >.<
Nowadays anytime numbers are involved I write the tests after doing manual tests multiple times and getting the expected response, and/or having an LLM check the work and make suggestions. That in itself introduces more issues sometimes since that can also be wrong. Probably should have paid attention in school all those years ago lol
Game dev seems like a place where testing is a bit less common due to need for fast iterations and prototyping, not to say it isn’t valuable.
I’ve seen a good talk (I think GDC?) on how the Talos Principle devs developed a tool to replay inputs for acceptance testing. I can’t seem to find the talk, but here is a demo of the tool.
The Factorio devs also have some testing discussions in their blog somewhere.
The Talos Principle video was interesting to watch, thanks for the link! It shined a little bit of light on automated testing.
Theres also someone on YouTube who has been teaching an AI on how to walk and solve puzzles on its own, the channel name escapes me and I’m nowhere near a working computer to look it up at the moment :(
Aw man, I can empathise. I don’t personally have any issues with mathsy stuff but I can imagine it being a huge brick wall at times, especially in game dev. I wish I had advice for that but its not a problem I’ve had to solve!
We’ve definitely written lots of tests that felt like net negative, and I think that’s part of what burned some devs out on testing. When I joined, the few tests we had were “read a huge JSON file, run it through everything, and assert seemingly random numbers match.” Not random, but the logic was so complex that the only sane way to update the tests when code changed was to rerun and copy the new output. (I suppose this is pretty similar to approval testing, which I do find useful for code areas that shouldn’t change frequently.)
Similar issue with integration tests mocking huge network requests. Either you assert the request body matches an expected one, and need to update that whenever the signature changes (fairly common). Or you ignore the body, but that feels much less useful of a test.
I agree unit tests are hard to mess up, which is why I mostly gravitate to them. And TDD is fun when I actually do it properly.
I hear you. When you’re trying to write one big test that verifies the whole code flow or whatever, it can be HELL, especially if the code has been written in a way that makes it difficult to write a robust test.
God, big mocks are the WORST. It might not be applicable in your case, but I far prefer doing some setup and teardown so that I’m actually making the network request, against some test endpoint that I set up in the setup stage. That way you know the issues aren’t cropping up due to some mocking nonsense going wrong.
Asserting that some arbitrary numbers match can be quite fragile, as I’m sure you’ve experienced. But if the code itsself had been written in such a way that you had an easier assertion to make, well, winner!
Its all easier said than done, of course, but your colleagues having given up on testing because they’re bad at it is kinda disheartening I bet. How are you gonna get good at it if you don’t do it! :D
I definitely deserve a lot of blame for designing my primary project in making hard to test. So, word to the wise (though it doesn’t take a genius to figure this out), don’t tell two fresh grads and a 1 YoE junior to “break the legacy app into microservices” with minimal oversight. If I did things again, I still think the only sane decision would be to cancel the project as soon as possible. x.x
I actually was using a mock webserver with the expected request/response, which sounds like what you’re getting at. Still felt fiddly though and doesn’t solve the huge mock data problem which is more an architecture design failing.
I’ve mostly gotten away from testing huge methods with a seemingly arbitrary numbers in favor of testing small methods with slightly less arbitrary numbers, which feels like a pretty big improvement.
True. :)
Hahahahaha I feel that re: just kill the project!
Ah I thought you were just mocking the response, as opposed to having some real webserver so you don’t have to faff with mocking stuff. Sounds like you did what I would have :P
That does sound like a big improvement! Anything you can do to make your own job easier