Yes, the modes in the band plan are mandatory, with some areas of ambiguity, some areas that are under-specified on the graphical band plan chart, and some very limited room for error.
For instance, the chart shows clear division between CW and Phone sections, but there is a note on the side that says that CW can be used anywhere except 60m. The key says red is CW and data, but it doesn't list all the data modes. There are digital modes (codec2) that are considered phone rather than data, so can't be in the red sections. Similarly, digital modes for images are not data and also can't be used in the red sections.
On the other hand, while there is a convention for where to use USB vs LSB, this is not so strict. But if you get it wrong, probably nobody will talk to you (or be able to demodulate your transmissions). The band plan is a mix of legal requirements and "gentlemans' agreements" on how to use the spectrum; while the latter is not required, failing to follow them will not make you friends or contacts. Communication is hard enough even when people do follow the conventions.
As to accidentally transmitting phone in the CW bands -- doing this could interfere with a large number of stations, since you can easily fit 10-30 digital stations in the space of one phone transmission. Doing it once briefly probably won't get noticed. Doing it for an extended period, intentional or not, could easily generate many complaints against you as you interfere with other stations. Using phone too close to the CW band edge might not generate many complaints, but you might find a Volunteer Monitor observation card in your mailbox, and repeated offenses could cause you to lose your amateur license and more. Transmitting in the extra sections with a general license is also very likely to get noticed.
Part of being an amateur radio operator is following good practices. If you are sloppy in following the band plan, you are probably being sloppy in other areas as well that could endanger the safety of you and those around you. You should make an effort to pay attention and do things correctly.