I have Windows 7 on Thinkpad. I can't connect FSMS to either a Hauwei EC168C "dongle" cell modem or an LG Shine CU 720 phone.
The dongle works fine for Internet and can send SMS via its own software.
In FrontlineSMS I get a "Detected" on COM3 but then a "Failed to connect" message.
Have you worked through the "Troubleshooting" section in the "Help" files? There's a whole section there which deals with connection issues.
Take a look there and see how you get on.