Windows service setup custom action Error 1001 System.BadImageFormatException

I had windows service and setup projects developed in VS 2008 .NET 3.5.

Setup was converted to VS 2010 and windows service project replaced with 开发者_如何学Cnew one created in VS 2010 .NET 4.0.

There are custom actions in setup to install service in the system.

The Error 1001 System.BadImageFormatException failed to load service exe due to incorrect format appears during setup execution.

Without custom action the setup runs without problems. I can also manually install the service with installutil.exe

Are there any settings in the setup project to run correct installutil from the right .NET version?