How to craft a persistent install USB media using common linux tools?

Without resorting to specialized tools like UNetbootin et al, how to create a USB install media (from a distro ISO file) that have a write (persistence) partition? I tried just creating two partitions and using dd on the first, but I cannot figure out how to have the boot work AND the partition table header. This is more about understanding the process and how things work than just accomplishing the task, so please, do not answer with another tool that is a do it all blackbox . But if there are utilities other than dd that help with specific steps, that is ok and welcome.

Question by gcb on Super User, licensed CC BY-SA. We show a short excerpt and link to the original thread as credit.