Friday, April 28, 2017

Tip: You Should Consider Purging the DBA_Recyclebin

The Oracle Recyclebin feature introduced with version 10G is somewhat analogous to deletions from Windows recycle bin functionality. Essentially Microsoft obfuscates the location of the file and also marks its storage as available for other use by the system. There are third-party tools which may allow you to recover them, although the probability of recovery declines over time as the system has need for available space for other purposes. In a similar way, when an authorized Oracle user drops objects like tables and dependent objects, like indexes and constraints, they aren't necessarily removed from storage (assuming a default recyclebin parameter ON/TRUE); they are essentially renamed with a BIN$ prefix. You can identify the current OBJECT_NAME from DBA_RECYCLEBIN using OWNER, ORIGINAL_NAME; you can recover by FLASHBACK TABLE owner.original_name TO BEFORE DROP. (Note that there are some nuances and exceptions to recycle bin functionality (e.g., recovering dropped schemas or objects of the SYS user) and limited recovery options.) Also note that the ability to recover dropped objects lessen over time as space is needed for other purposes (e.g., inserts into same tablespace tables or user space quotas).

Oddly enough, the recycle bin issue came into play in an indirect way. To provide a context, the virtual container recycle bin will grow over time. You can control this in various ways, e.g., turning the initialization parameter off or executing the purge command.

In my current position, we often do selective schema exports (in addition to regularly scheduled exports or database backups) on request for development. These exports are scripted to the extent that fellow privileged administrators with limited Oracle background can run them. We also administer multiple databases that are similarly designed and structured, and the export task is repeated across database servers.

The wall time for the 2-schema export task run manually is for most servers within 15 minutes [Note that in this context, we are discussing data pump, i.e., expdmp, not exp.]. One day a system administrator ran the ad hoc task request on a third database instance when the export of the second schema seemed to hang for literally 3.5 hours. I analyzed the alert log; I tried export alternatives (if I excluded statistics and the table filter in the export parfile, I could do the export in less than 15 minutes). Development wasn't happy with the delays and pointed out no other DBA's were reporting similar issues with their servers; it must be local administrator "incompetence". It wasn't obvious why this server was behaving differently from the others; they had similar configurations, applied patches, etc.

I currently don't access to Metalink/My Oracle Support; I tried to search on abnormally long exports or statistics using Internet search. (Why did I pursue that? Because it seemed to be hanging on the initialization stage referencing statistics: ") Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS")  But most of the problems I found focusing on statistics were on the import, not the export stage.

A DBA colleague had stumbled across a discussion of dba_recyclebin. I don't know which source he found, but I have found this discussion on data migration using data pump which advises (without explanation) purging the DBA_recyclebin before running expdp.

What we found is literally thousands of objects in the recyclebin, the vast majority of which existed in the problem schema. My colleague ran purge dba_recyclebin from the internal account as sysdba, and the proof is in the pudding: the subsequent wall time for the problem schema export shrank to about 3 minutes.

It's still not clear why the dropped object problem was peculiar to this database instance given standard installation, patching and scripted application functions. But if you don't disable the recyclebin default, at least create and schedule periodically  a purge table script from DBA_RECYCLEBIN (e.g., using droptime filter). Chances are, you can't count on being able to recover long-dropped objects anyway.