This design not only shows you how to use ASP.NET UpdatePanel inside ModalPopup, it also shows a way of having multiple dynamically created controls that act as triggers to show ModalPopup, and the content of ModalPopup will change depending on the control that triggered the Popup. Then user can perform an action that does partial update inside the ModalPopup. You will only need one ModalPopupExtender control by using this design.

The following diagram shows how the controls are structured in the .aspx page

As you can see here, we have an UpdatePanel inside the ModalPopup, and the ModalPopup is in the another UpdatePanel along with other dynamic controls on page.

The next diagram shows the actual look of the page.

More...