Saturday, November 11, 2017

Practical Tips I: Compressing Files

Quite often in a consulting scenario I'll find myself in a situation, particularly working for the government, where I'm dealing with tight constraints and need to find space. I remember when I was sitting on several gigabytes of archived logs going back 4 years. I think the oldest backup image datafile (yes, the client wasn't running RMAN) was maybe 6 months old (now, in practice, I've never gone past since my last usable backup, even for doing a clone or refresh of test or development), but the point is that even if the client wanted to clone the production database as of 7 months ago, I didn't have the necessary datafiles. But I couldn't get a client manager to give me the OK to purge 3.5 years of unusable archive logs.

In a recent situation, I needed to find enough space on my /u01 and /opt mountpoints to make the prerequisite storage requirements for running a major bundle patch. Here's one tip: look for audit files, something other DBA's don't realize until one day they find themselves facing a full file system. One source is to check your admin folders for adump, e.g., /u01/app/oracle/admin/[SID]/adump. If you're running on a grid (clusterware/RAC), also look for an audit directory off $GRID_HOME/rdbms.

So what do you do when you run into wildcard problems because you have too many files in your directory? (I'm assuming here that you are running on Linux or a Unix variant.)  I love to do DO LOOPs; there are a variety of things you can do (including moving files where you have more space), but if you delete files after zipping them, you should reclaim available space throughout the process. So, for example, assuming you are running in the audit file working directory

ls | while read fn
do
zip aud.zip $fn
rm $fn
done

Now there is a complication in Linux: what if you have say a file name preceded by a minus sign, e.g., like I've seen in grid audit files. A practical solution is to work "-- "" into the command syntax, e.g., ls -- *.aud

I'm a fan of using the job scheduler in OEM 12c. (I find the interface design somewhat quirky and non-intuitive to launch job creation off the bottom of the screen.)  Now one of the things I frequently like to is date-stamp a file (say, for instance I want to run a periodic [weekly] audit file sweep indicating the end of the audit file period), e.g.,

cd [audit file directory]
export zf=aud`date +%y%m%d`.zip
zip $zf    *.aud
rm *.aud


Now this script will not work (in my experience) through OEM job scheduler, and it has to do with the use of %. So what you want to do is to repeat the '%', e.g.,

export zf=aud`date +%%y%%m%%d`.zip

Monday, May 1, 2017

Why I Became an Oracle DBA

I've made my living as an Oracle DBA since 1993. I've often had auxiliary responsibilities, e.g., development, tech lead, ERP administration and system administration, but my primary responsibility has been as a DBA. Did I grow up with the ambition of being a DBA? No. I grew up in the era of mainframe computing. In fact, I had to use card punch machines to write my first computer programs (FORTRAN) in college; My college didn't have its own computer, so I had to submit my card decks to a sister campus in San Antonio. (There was a 3-day turnaround, so I could only afford to make one mistake for the weekly assignment.)

Originally I thought I  had a vocation for the Roman Catholic priesthood, which is why I attended OLL, a private college. (I was an Air Force brat, and my family at the time lived at a military installation about 150 miles south of San Antonio. I didn't have a car or a license.) I didn't necessarily want to be a diocesan priest, like my maternal uncle. I had done well in school and naturally thought about teaching as a career. I did think of joining a teaching order of the Church, e.g., the Jesuits or the Oblates. In fact, I had a preliminary interview with the Jesuits at my college dorm; I don't think they ever followed up.  Originally, I had intended to major in secondary education and probably would be a math/science high school teacher in Texas, except I fought my academic adviser to get into a philosophy class. (My adviser wanted me to take art history; but philosophy is a common major for prospective seminarians.)  That course changed my life; philosophy remains my one true love. I transitioned to a double major (in math and philosophy). But philosophy is not a core reference discipline for secondary education. More to the point I had progressed from wanting to be a teacher to becoming a professor. There was the one experience that cemented the idea in my mind; I went to visit one of my principal philosophy professors, an Oblate priest, in his office. I walked into a room where he was probably playing his favorite, Leonard Bernstein, on a record player, pungent pipe smoke filled the room, walls crowded with books: making a living talking about deep thoughts sounded like heaven on earth.

Unfortunately, beyond teaching, there aren't many professional opportunities for philosophers, and job opportunities for philosophy PhD's were extremely limited and competitive--basically one had to hope for existing tenured faculty to resign or retire. (A number of people had frank conversations with me about the marketability of my philosophy degree.) So I shifted to consider becoming a math professor, In fact, one of the RC sisters/nuns who had earned her degree at a Louisiana university had lobbied to get me considered for its PhD program; I was verbally promised a graduate student financial aid package, but to be honest, I had my sights set on a more prestigious program at the University of Texas at Austin (where my academic adviser had earned her PhD). Another case  of woulda, coulda, shoulda. I almost instantly got into political trouble as a teaching assistant to a visiting East European professor from Ohio State. (The students were at a point of rebellion; he was difficult to follow and had unorthodox tests. As a teaching assistant, I had 2 problem solving class session classes, and the students were basically unhappy with me as a surrogate to the professor, feeling maybe I wasn't doing enough to prepare them for his exams; to me, the definitive moment was when a coed, the best student in class, literally threw her second exam in my face as she left the room. (It's not like I knew what he was going to ask on the exam.) Another classic moment: one day he starts writing on a series of blackboards in the front and then on the right side of class. The students were just mechanically just jotting down everything he was writing; he wasn't really lecturing; he was just filling up the boards without comment. Suddenly, about 8 boards into his writing, he seemed to get lost in whatever he was trying to do; he looked up at me and sheepishly asked if I knew where he was going with this. (I didn't.)

The other TA set me up, saying we had to have an intervention. We agreed I would start. The professor was troubled by what I had to say and asked Jeff for his comment. Jeff turned on me, effectively saying, "Yeah, a few students are complaining, but you get some of those malcontents in any class." What the hell? I had no idea why Jeff had set me up like that, but the professor decided that I was the one with an evil agenda and became obsessed with me. I'll never forgot how he followed me into my graduate real analysis class and started bitching about me to the professor. It then got even worse; I got called to the department chair's office where I got yelled at and threatened over my "disloyal" actions. I think I knew at that moment I would never get my terminal degree from UT and would have to settle for my MA.

There were other considerations, too. It turns out there was a glut of mathematicians on the market and newly minted PhD's, if they could find a job, would have to settle for some obscure state branch university. So they decided to raise doctoral qualification to 6 exams with limited retries and a time table. I had befriended a married couple in the department; she got through her 6, but her husband was dropped from the program lacking the final sixth exam. (I never really sat for the exams, but they had become a moot point: during the spring semester it turned out I had lost my teaching assistant position for the next year, and I really couldn't afford to go to school. I got a token grader job in my last semester working on my Master's thesis, but I had to ask my maternal grandfather for a $500 loan so I could have it professional typed to department standards. (This was in the days of typing, and you needed specialized equipment to print mathematical equations.)

My job prospects were limited. I originally thought that the Air Force was going to offer me a commission and have me trained as a meteorologist, but I was passed over in the selection pools. There were no local community college instructor jobs. I tried to check into trying to pick up high school teaching credentials, but I didn't have the resources.

Now most math majors at the time almost inevitably minored in computer science, but UT had an excellent philosophy program, and it was a no-brainer at the time. Now it was a lost opportunity, as I found out a pure (vs. applied) math degree wasn't that marketable. Long story short, I eventually got a computer programming trainee position at a prominent insurance company in San Antonio. They were having problems training other programmers (say, COBOL) to write in APL, a mathematically notated interactive language that IBM had developed for rapid prototyping; APL was often used in areas like property actuary at the insurance company. There's another story behind that experience; let's just say I taught myself APL well enough to attract a job offer with a substantial increase in salary the next year from the largest APL timesharing vendor in Houston. (The timesharing business model was to sell premium-priced computer time running customized enhanced APL applications; this was a time when buying another mainframe meant big bucks, plus it was a workaround to in-house application backlogs.) The introduction of the PC was a disruptive technology; by the time I earned my MIS PhD in the mid-80's, the companies I had worked for had all gone out of business.

After about a year of living in Houston, I started looking at picking up an MBA at the University of Houston, which had a part-time nighttime program which I could fit into my work schedule. (I also hoped that it might improve my dating prospects; in fact, I did date a few coeds but nothing serious.) By my second year I was aware that the business school had a PhD program in MIS, which reopened my deferred dream of becoming a professor, was admitted and eventually got a part-time teaching stipend (2 classes a semester), enabling me to study full-time.

One of the attractions of the discipline was MIS classes were "hot"; I used to joke that we could offer "Basketweaving for MIS" and hit a class limit of 35 in 10 minutes of registration. Business schools desperately wanted newly-minted PhDs; granted, the land grant name schools were highly competitive (our first 2 graduates went to Arizona State and Iowa State), but when I first hit the academic market in the mid-1980's at conferences associated with two competitive doctoral consortia, I literally did not have to dial a phone; I fully booked every 30-minute slot available and probably could have booked at least 2-3 more days of interviews. That doesn't mean I got dozens of campus visit offers (prerequisite to a job offer; it usually involves a research presentation and several potential colleague/administrator interviews. I usually got 2-4 campus visits while I was on the market and 0-2 offers.)

I had started my MIS doctoral coursework by the time I had finally finished up the MBA in 1983; I had taken two database classes under Richard Scamell, easily one of the best professors I've ever had (and my future dissertation chair). Scamell had a contact at Oracle; this was around the time of Oracle 2, when documentation came in three-ring binders and SQL*PLUS was called "UFI" (user friendly interface). UH was running Oracle on an IBM-compatible mainframe. What I particularly remember is that we had a hard time doing our database projects due to lousy uptime; in fact, the entire class had to take incompletes that fall semester, I think that's the first time I had really encountered DBA issues or was familiar with that role. I don't think at the time I knew it would become a future career; recall, I was convinced I would a professor until retirement. Still, of all the roles I would deal with as a professor (systems analyst, etc.), a DBA role had the most appeal.

My dissertation was not on database, but my research interests included human-computer interaction, including the use of SQL, the de facto standard query language behind relational databases, like Oracle, DB2 and SQL Server. I also had a focus on documentation. One of my office mates back at UH, Minnie Yen, did look at query languages in her dissertation, so around the turn of the 1990's as my academic career began to wind down, I had proposed a joint project, my writing a "minimal manual" on learning SQL for a research project including one or more of Minnie's students.

I didn't get a chance to teach database during my 3 years at UWM, because 2 of my tenured colleagues who did more technical database research "owned" the courses (although I seem to recall they were using popular PC data applications, not SQL. I think while I was there Oracle had decided to roll out a PC bundle for Oracle 5 for roughly $200 (not that bad considering you got a huge stack of paper documentation with the bundle), and I bought a copy; however, I don't think Oracle was installed at any of the 3 universities I taught at as a professor.

I finally got a chance to teach database (and, separately, data structures) at UTEP; we didn't have Oracle, but I was determined for my students to have exposure to SQL--and used an option similar to when I taught structured 1985-standard COBOL using textbook bundled Ryan-McFarland software. In this case, the textbook I chose came bundled with a limited version of XDB-SQL.

UTEP had hired me principally because they were up for AACSB accreditation and wanted my research vita/resume at file (I'm not saying that I was a deciding factor, but I would be a second published MIS researcher in the department), and I soon discovered that the department chair (it was a fusion department, including production logistics faculty) quickly reneged on key promises and probably the most obnoxious department chair I've ever met. I'm not going to go into specifics here other than my stay in El Paso was a hellish experience, so much so I went on a campus visit in Louisiana during finals week of that first fall semester. (It was not my idea; I wanted to defer it to the spring. For some reason, they refused  And then they decided not to make an offer, which I would have accepted.) By the spring, it was clear that UTEP wasn't that thrilled with me, either. My options were limited by that time because most of the colleges had done their screenings at the late fall conferences. It turned out that Illinois State had a temporary 1-year appointment available because one of their applied computer science professors was trying to set up a research facility.

As in the other two universities, there were some nasty office politics at play. I went on the market, but I found that the market had drastically changed. There were probably 3 reasons: (1) to help the regional schools unable to hire new MIS PhDs, Indiana University offered a summer program to retrain other professors (e.g., PLM, QMS) to MIS,  (2) there was a bumper crop of new PhD's; for example, I had leapfrogged 12 ABD's to graduate #4 (#3 defended her dissertation one week before me); almost all of these went through as UH warned them they might need to re-qualify as their 5-year clock ended, plus others I had studied with joined the market; (3) MIS class enrollments started to cool off. I had noticed this at UWM, where classes of 35 the next year had available slots, like 28 enrolled. Also, there was a recession, and even computer science graduates, who normally didn't want anything to do with business schools, were submitting their vitae.

The market had radically shifted; schools that normally got a dozen resumes and hoped their top 3 would be available suddenly got flooded with 85 resumes, so then they quickly had to adapt. It depended on the school but they developed filters which didn't necessarily favor me, despite my research and teaching record, not being an ABD. Maybe, for instance, a male-dominated department might want to recruit a female candidate (I know one school that did this), there was a demand for computer networking faculty, etc.

I think I did get 2-3 campus visits, but no offers. Ironically, Illinois State was in the market, mostly to cover the service course (I was working for a hybrid Applied Computer Science department). Initially they wanted to talk to me, but I expect that some campus politics came into play, and they refused to return my calls.

I didn't realize it at the time but my academic career was over. I did get an offer in 1994, but the offer came several weeks after my visit, and I had just started a new job; a lot of professional recruiters were worried that I wanted to use them until the academic market recovered.  I was between jobs when I went on the campus visit, and I made it in good faith. They had led me to believe that I would get a status in 2 weeks and that didn't happen. (Of course, it's possible that they ran into bureaucratic inertia) It certainly wasn't the money; they were offering the highest figure I ever got from a school and more (by over $10,000) than I was making in my new job. This would be the last time I got a serious offer. I came close to joining the faculty of a Catholic college in the DC metro area a few years later. That, ironically, was probably my biggest disappointment; I have a BA from a Catholic college; I went on campus visits to 4 Catholic colleges and no offers. It wasn't money; for example, Providence College made it clear the max they would pay is $35K, far lower than my state-sponsored offers.

So looking for work, I had to look back at returning as an IT professional. (An industry research position was not an option; for example, IBM at that time was struggling.) As mentioned above, a lot of IT recruiters saw me as trying to use the company until an academic offer came along; it didn't matter what I said. Others discounted academic computing experience and considered my prior job skills as probably eroded. Still others refused to consider me for anything beyond entry-level work.

I finally found some APL work at an IBM subsidiary in the Dallas area, but I was really focusing on reinventing myself as an Oracle DBA. The problem was chicken or the egg in terms of experience. Finally, an EPA contractor was willing to offer me a position at the Chicago regional headquarters. They paid me peanuts, but I got my foot in the door. On literally my second day on the job, they lost their recompete on the contract (which basically meant I would be looking for a new job in a few months). I got my first extended gig at a private company which is now an Equifax subsidiary.  I spent 2.5 years there (and would probably still be there if the owners didn't sell). [The CEO worried that Accenture was a threat to their business model.] About a year into the job, I got unsolicited calls from a Coopers & Lybrand recruiter helping set up an Oracle Apps/EBS practice. I essentially transitioned from an in-house DBA to a consultant by 1997 and have mostly served in a contractor role since 2004.  I may expand on my experience in future posts, although my first blog post features some insights from my specialty as an Oracle Apps DBA.

Friday, April 28, 2017

Tip: You Should Consider Purging the DBA_Recyclebin

The Oracle Recyclebin feature introduced with version 10G is somewhat analogous to deletions from Windows recycle bin functionality. Essentially Microsoft obfuscates the location of the file and also marks its storage as available for other use by the system. There are third-party tools which may allow you to recover them, although the probability of recovery declines over time as the system has need for available space for other purposes. In a similar way, when an authorized Oracle user drops objects like tables and dependent objects, like indexes and constraints, they aren't necessarily removed from storage (assuming a default recyclebin parameter ON/TRUE); they are essentially renamed with a BIN$ prefix. You can identify the current OBJECT_NAME from DBA_RECYCLEBIN using OWNER, ORIGINAL_NAME; you can recover by FLASHBACK TABLE owner.original_name TO BEFORE DROP. (Note that there are some nuances and exceptions to recycle bin functionality (e.g., recovering dropped schemas or objects of the SYS user) and limited recovery options.) Also note that the ability to recover dropped objects lessen over time as space is needed for other purposes (e.g., inserts into same tablespace tables or user space quotas).

Oddly enough, the recycle bin issue came into play in an indirect way. To provide a context, the virtual container recycle bin will grow over time. You can control this in various ways, e.g., turning the initialization parameter off or executing the purge command.

In my current position, we often do selective schema exports (in addition to regularly scheduled exports or database backups) on request for development. These exports are scripted to the extent that fellow privileged administrators with limited Oracle background can run them. We also administer multiple databases that are similarly designed and structured, and the export task is repeated across database servers.

The wall time for the 2-schema export task run manually is for most servers within 15 minutes [Note that in this context, we are discussing data pump, i.e., expdmp, not exp.]. One day a system administrator ran the ad hoc task request on a third database instance when the export of the second schema seemed to hang for literally 3.5 hours. I analyzed the alert log; I tried export alternatives (if I excluded statistics and the table filter in the export parfile, I could do the export in less than 15 minutes). Development wasn't happy with the delays and pointed out no other DBA's were reporting similar issues with their servers; it must be local administrator "incompetence". It wasn't obvious why this server was behaving differently from the others; they had similar configurations, applied patches, etc.

I currently don't access to Metalink/My Oracle Support; I tried to search on abnormally long exports or statistics using Internet search. (Why did I pursue that? Because it seemed to be hanging on the initialization stage referencing statistics: ") Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS")  But most of the problems I found focusing on statistics were on the import, not the export stage.

A DBA colleague had stumbled across a discussion of dba_recyclebin. I don't know which source he found, but I have found this discussion on data migration using data pump which advises (without explanation) purging the DBA_recyclebin before running expdp.

What we found is literally thousands of objects in the recyclebin, the vast majority of which existed in the problem schema. My colleague ran purge dba_recyclebin from the internal account as sysdba, and the proof is in the pudding: the subsequent wall time for the problem schema export shrank to about 3 minutes.

It's still not clear why the dropped object problem was peculiar to this database instance given standard installation, patching and scripted application functions. But if you don't disable the recyclebin default, at least create and schedule periodically  a purge table script from DBA_RECYCLEBIN (e.g., using droptime filter). Chances are, you can't count on being able to recover long-dropped objects anyway.