My two cents.
To me, Heat is another tool in the box that builds the fiction. The mechanical events that happen via entanglements are fine and can be hand waived most of the time, but each entanglement should cause ripple effects that stick around. Just because the crew paid off the Blue Coats doesn’t mean that that file sitting on a desk somewhere just goes away. I always ask myself who the mechanic affected and how it relates to their disposition towards the crew. Sure, maybe the crew paid off the Blue Coats and now they think they’re in the clear, but maybe they pissed off a couple of coppers during the transaction, or maybe those same cop partners see an opportunity to bleed the crew for even more coin down the line. Give those coppers names and viola, two new NPC’s I can use to put pressure on the players.
And, of course, entanglements don’t just apply to Blue Coats. They can relate to other criminal factions, powerful individuals and supernatural entities. Sooner or later they are going to catch the attention of a demon. Sure, they’ll probably just satisfy the mechanic that takes care of the initial entanglement, but that demon doesn’t just go away. The crew is on it’s radar now, and that is pure plot gold to me.
The Heat system is as abstract and boring as you want it to be. Every entanglement has the option of simply paying the coin or losing the rep, or what have you, OR you can zoom in on the situation and role-play it out, which generates NPC allies and enemies, all with their own agendas and potential axes to grind.
As SymbolicCity pointed out, it sounds like your players are very cautious and do everything they can to avoid heat. So it’s your job to put them in situations that turn that heat up (so to speak). One of the easiest Devil’s Bargains to offer up is a simple +2 Heat. They should be pulling scores against high profile and well connected targets (+1 Heat). They should be pulling off scores in hostile territory (+1 Heat). And remember, if killing was involved in a score -even if they didn’t do the killing- that’s another +2 Heat. And if you can push them into a war with another faction, not only do they take +1 Heat during payoff, but they each only get a single free downtime activity. And you don’t have to be the mean, manipulative GM to put them into these positions. The game does that for you. The system itself generates complications, any one of which can push a PC towards killing a guard or the crew angering another faction to the point of war.
You wondered if maybe the economy should be toned down to 2-4 coin per score. Honestly, I think the suggested book payoff for scores is a little stingy. I regularly pay out 8-10 coin for a score and my PC’s still end up having to take money out of stash to pay for more downtime activities. In my own recent post I talk about my current crew having ridiculous amounts of coin. Recently they earned 48 coin in a single session from two scores and their coin generating claims! With four players that came to 12 coin per player! And yet they still wound up spending almost all of it to recover wounds, get rid of stress, replace cohorts, get rid of Heat, and work on about a dozen long term project related clocks they have going on.
My point is, the more you complicate their lives, the more things they have to spend coin on, and the less easy it becomes to simply knock off all their Heat. And if you can really take advantage of when things go sideways during a score it should be relatively common for their Heat during payoff to shoot into a wanted level. At least that’s how it’s been for my players, although I will admit that they are generally a bloodthirsty bunch and have no qualms about using high explosives and leaving bodies in their wake, so obviously YMMV.
Finally, a word about Doskvol law enforcement. I think of LE in Doskvol as having three official branches: The Blue Coats, the Inspectors, and the Spirit Wardens. I think of most entanglements relating directly to the Blue Coats. They are the ones who are corrupt and easy to pay off. The Inspectors, on the other hand, are famously incorruptible, and that’s where things get interesting. Even if your crew is regularly paying off the Blue Coats and keeping their wanted level at zero, they are still committing crimes, and it doesn’t require an entanglement roll for an Inspector to take an interest in that.
For example: One of my crew carried out an assassination at a masquerade ball. While the PC doing the actual killing was strangling the mark he took a Devil’s Bargain of +2 Heat. In the fiction I equated that with the victim pulling a button off of the PC’s coat. The victim was a noble, so the Inspectors get involved in the investigation and the lead detective finds the button clutched in the dead woman’s hand. I named that Inspector, Drochack, and modeled him off of Beretta from the old show of the same name. Drochack is like a dog with a bone, and so now all of a sudden he’s showing up at random times to ask a PC “Just a few more questions; it’ll just take a moment of your time.” It got to the point where he had my crew seriously spooked. They couldn’t just kill him for fear of bringing down the wrath of the Inspectors as a faction, and they couldn’t just pay him off, so they were constantly working on clocks to frame someone else or otherwise throw him off their trail. The point is, law enforcement exists outside of the Heat system and you can use it to your hearts content. The same holds true for the Spirit Wardens. If your crew is a cult, it’s probably only a matter of time before they draw the attention of the Spirit Wardens, and that can bring on no end of headaches.
To answer your question: Should the Heat system incentivize the players to keep their wanted level as low as possible? I think the answer is, yes. But the overall game system tends to make doing that very difficult.
Should going to prison come with bigger upsides? I don’t know that there are ANY real upsides to going to prison. Usually it means sending some cohorts to Iron Hook to take the fall for the crew, which can sting, or it can mean one of the PC’s taking an extended vacation. I think it’s meant to be an undesirable thing, but it’s the price of doing criminal business in the Doskvol underworld.
Anyway, I’m sorry for going on and on. I just found your question very interesting. Good luck!