This is a popular thing to do in the web world, but you can do it by simply creating a gif image that is a rounded rectangle. Then you apply it to a table's properties as a border image. The Notes client will overlay this image over the borders of your table and you magically get a rounded table displayed in the Notes client.
This is covered in several blogs:
- nsftools
- Geniisoft
- BizzyBee