Specialized hardware accelerators, including third-party accelerators, can offer system designers high performance and/or low energy. They are increasingly complex and have interesting new abilities, such as sharing a unified memory space with the host system. However, accelerators may contain security vulnerabilities due to design flaws or malicious intent. I will give an overview of some types of threats that may be especially important when considering accelerators. I will then talk about a low-overhead approach to mitigating one particular flaw: incorrect memory accesses to host system memory by the accelerator. We show how accelerators can be forced to respect process memory access permissions as stored in the page table, with a per-accelerator storage overhead of 0.006% the size of physical memory and low (on average, 0.15%) performance overheads.