For two vector spaces V,W over a field F, then Hom(V,W) denotes the space of linear maps from V to W. Its dimension equals dimV×dimW. Note that Hom(V,V) is the endomorphism ring End(V) and that Hom(V,F) is the dual space of V.
The notation is more generally used; for example Hom(G,H), the additive group of homomorphisms where G,H are abelian groups, or to denote Hom functors in category theory.