Looking back at the pics now I think I was wrong. You can see Joseph's Cave in your first photo, it's under the white two story building to the left. I think the divers were next door at Pirates Cave which is under Pushcart. I think you can only get to Pirates Cave from the sea now. You can still go down through the blow hole on Miss Mae's place to get to Joseph's.
