Some people might suggest the nautilus as the ancestor of modern squid, since they're the only "ancient-looking" cephalopods most of us have seen. But the lineage that led to modern nautiluses and the lineage that led to modern squid diverged long before there was anything that looked like a squid.

Here is a tree of evolutionary relationships within the Cephalopoda, from the magnificent Tree of Life website.

Others might claim ammonites as the ancestors of squid, but as is clear from the above tree, they're not a much better candidate than nautilus. Ammonites and coleoids were on different trajectories, going way back.

So let's take a look at the coleoids. Here's how ToL breaks them down:

  • Belemnoidea

  • Neocoleoidea
    • Decapodiformes Leach, 1817 (squids, cuttlefishes and their relatives)

    • Octopodiformes Berthold and Engeser, 1987 (vampire squid and octopods)

The belemnites are exctinct, and I have been known to point to them as the ancestors of modern squid. I think it's reasonable to suggest that the ancestors of modern squid were probably something similar to a belemnite . . . which were, themselves, actually more similar to modern cuttlefish than modern squid.
When cephalopods first evolved, the primary feature that separated them from other molluscs was a shell with buoyancy chambers. This allowed cephalopods to invade the pelagic realm where predators and competitors were few. When the shell became internal in coleoids the bouyancy chambers were retained in belemnoids. In most present day neocoleoids, the buoyant shell has been abandoned (it is retained in Spirula and the Sepiidae) and many squids and octopods are negatively buoyant. While this is not a problem for those octopods that are benthic, negatively buoyant squids, however, must swim constantly to stay afloat.