Yeah, if words were actually encoded as 1-hot vectors this would be pretty trivial, but the rest of LLM training would be somewhere between infeasible and impossible. The actual embedding vectors obscure spelling even more.
Side note: last time I checked, current embedding vectors were approximately 40 dimensional… Has that gone up significantly in the last couple of years?
Semantic Vectors don’t work that way.
Yeah, if words were actually encoded as 1-hot vectors this would be pretty trivial, but the rest of LLM training would be somewhere between infeasible and impossible. The actual embedding vectors obscure spelling even more.
Side note: last time I checked, current embedding vectors were approximately 40 dimensional… Has that gone up significantly in the last couple of years?
A fair bit. EmbeddingGemma is open weights and allows for 128-768 dimensions.
It’s not as simple as more dimensions = better, due to size, efficiency, and context rot limitations though.
Introducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings - Google Developers Blog - https://developers.googleblog.com/en/introducing-embeddinggemma/