Mapping Center Answer:

One of the options, when inserting a neatline in ArcMap is to round the corners, which is probably the simplest of all the things you might do. The general approach for a border that is a bit fancier is to integrate marker symbols at the corners with one or more lines connecting them. Typically you need to have a marker symbol that is symmetrical and it will need to be rotated specially for each corner. The marker symbol would be drawn with a graphic point that you add using the New Marker tool on the Draw toolbar.

For constructing these more advanced neatlines, check into using either the snapping grid or the layout guides to make easier work of getting the markers to exactly align to the connecting lines of the neatline/border. Another set of tools is on the graphics toolbar. Finally you may also find using the arrow keys useful to nudge the selected graphics into postion.

A note about designing your map relative to fancy borders or neatlines: Take care not to make the border/neatline the most interesting or beautiful thing on the page. Your map, not its frame, needs to be the center of attention and what people notice first and focus on.