Reposting from my comment https://lemmy.world/post/37758804/20109240 which I recommend to check, as someone did a test with Dark Souls 1 and IMHO was unsurprisingly disappointing, namely it does recognize the game (honestly, not bad) and get the right boss (which name is literally on screen) and make kind of sometimes useful suggestions. But like… what’s the point? Who would play a game and… NOT know its name? Or not be able to search based on a boss name or a weapon name with existing dedicated good online guides?
Anyway… if you still want to try yourself WITHOUT relying on Microsoft consider :
"If someone somehow wants to test this locally I suggest
install locally a vision model, e.g. Moondream (which Ollama supports but alternatives too), then
take a screenshot of your game,
write a prompt like “How can I play this game better”
query the vision model with the image and your prompt
marvel at how pointless and costly the whole setup is and how a basic query on e.g. DuckDuckGo with “game name” + prompt would yield way WAY better results from actual human, uninstall the whole, keep on playing with your actual brain.
At least now you can say you tried before you complain, rightfully, that it sucks.
PS: I didn’t actually try this, I’m too lazy for that right how but feel free to report back if you do!
Edit : 2 potential optimization (despite not being sure it ever makes sense in the first place!)
do so automatically, e.g. ~/gaming_screenshots directory (via e.g. Spectacle shortcut) monitored via inotify then notify-send the suggestion, thus stay in game during the whole process
If there was no user interaction needed I’d agree but here, AFAICT, the user still needs to ask. If it’s truly entirely pointless then once the novelty fades away it won’t be used which deprived of precious data points, namely user feedback. For non interactive use I imagine streaming provides already a lot.
Reposting from my comment https://lemmy.world/post/37758804/20109240 which I recommend to check, as someone did a test with Dark Souls 1 and IMHO was unsurprisingly disappointing, namely it does recognize the game (honestly, not bad) and get the right boss (which name is literally on screen) and make kind of sometimes useful suggestions. But like… what’s the point? Who would play a game and… NOT know its name? Or not be able to search based on a boss name or a weapon name with existing dedicated good online guides?
Anyway… if you still want to try yourself WITHOUT relying on Microsoft consider :
"If someone somehow wants to test this locally I suggest
marvel at how pointless and costly the whole setup is and how a basic query on e.g. DuckDuckGo with “game name” + prompt would yield way WAY better results from actual human, uninstall the whole, keep on playing with your actual brain.
At least now you can say you tried before you complain, rightfully, that it sucks.
For more check https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence
PS: I didn’t actually try this, I’m too lazy for that right how but feel free to report back if you do!
Edit : 2 potential optimization (despite not being sure it ever makes sense in the first place!)
Spectacleshortcut) monitored viainotifythennotify-sendthe suggestion, thus stay in game during the whole process" and again feel free to share back results.
I don’t think the point is to be useful, but more to collect data on how you play and how to play.
If there was no user interaction needed I’d agree but here, AFAICT, the user still needs to ask. If it’s truly entirely pointless then once the novelty fades away it won’t be used which deprived of precious data points, namely user feedback. For non interactive use I imagine streaming provides already a lot.