This summer is ORA600's 5th anniversary - and DUDE's 10th !
So I thought it would be appropriate to look back and reflect on all crazy and weird things we've gone through.
As many of you know, DUDE started out as jDUL on sourceforge. I started on it while I was working in South Africa - me and my buddy Kugendran Naidoo came up with all kinds of crazy idea's whilst enjoying a smoke in the smoking room. Wow, remember the days you could still smoke in the office ?
I can't say the'Oracle community' was very appreciative to the idea of a DUL-like tool. Back in the day, the Oracle community, for me, consisted mostly of the cdos newsgroup (comp.database.oracle.server
). To be honest, I found cdos quite a hostile place - a lot of flames, rants, newbie bashing etc… all in all sending out negative vibes. Thinking about it, it's just the opposite of the Oracle-l mailing list (http://www.freelists.org/archive/oracle-l
). I'm not surprised cdos is now full of spam messages and almost completely abandoned. If it wasn't for cdos … I might have gone through open-sourcing jDUL/DUDE !
Anyway - flash forward to the summer of 2005 - after a meeting with Mogens Norgaard from Miracle AS, ORA600 is born. (see also here
). That particular summer was quite fruitful as I spent most of my time implementing new features. To give you an idea - in 2001 the source code was 131Kb in size and 4393 lines long. By the end of 2005 it was 12390 lines of code and 446Kb in size. The current version is 1049Kb and 27817 lines long. I must admit, a lot of that is code documentation !
Although Miracle AS worked as an incubator, Mogens introduced me to Daniel Fink in 2007 at a Miracle Scotland event in Edinburgh. Daniel was interested in representing DUDE in the US. Mogens didn't mind. Dan is a great guy to work with - very professional! After that I started collaborating with different companies around the globe, like Pythian
, Evdbt Inc
(Tim Gorman), NRG Consulting
. I really think local support is very important for potential customers even if it squeezes my profit margins.
I have to admit - I have encountered some amazing corruption cases over the last 5 years. Some were solved fast and swiftly - others have costs me several years of my life
Here are some recovery cases I will never remember :
1. The one with the mixed data in a LONG column
This case was *by far* the most stressful recovery I have done. The customer was a newspaper publisher and they needed their database online before Sunday 2PM. Having a deadline is always stressful, but this database contained a table with a LONG column that was on average a gigabyte large. Normally that's not a problem, but at that time I had a bug in my DMP API related to Java's garbage collector, that caused the unload to slow down …. It actually appeared to be hanging as there were no error messages being generated. I didn't know about the bug right away so I investigated the situation up to the point I opened several blocks in a hexeditor, following pointers of chained rows manually (did I mention the deadline?). What I saw was that the column contained readable text and then suddenly what looked like binary data. I immediately thought some blocks were partially corrupted and what I was seeing was garbage. Quite often on windows platforms you'll see file corruption, where files are partially overwritten by junk in multiples of 512bytes, spread all over. It's like someone takes a machinegun and puts some holes in the files. Worst case, the corruption is in the middle of a block so header and tail are fine and the corruption goes almost unnoticed. Until you try to trail the row directory to the row headers and you suddenly hit junk. It gets even worse, if by coincidence the header is identified as a chained row and thus you try to read the next dba…. Which of course will point to some non-existing datafile and offset!!!
Anyway that's what it look like at first glance … until I actually started to put things together with an hex-editor (try doing that at 3AM at night!). The blocks were fine, the row directory was valid, row headers were ok, chained row headers were pointing to valid row pieces. This led me to the conclusion the data was valid, just extremely weird !
At that point I focused on my DMP API and found out the garbage collector was doing too much work, cleaning up after processing 1Gb columns.
Once I fixed that, the unload was blazing fast - and I made the deadline… but boy… what an adrenaline rush !!!
2. The one with the even larger column
DUDE is written in java - most JVM's are/were 32bit. Although DUDE is multithreaded (it has producer and consumer threads) and some platforms turn threads into procs, I have to work with a 2Gb memory limitation in mind. At one point I had a case were DNA strands were being stored in CLOBs. These were all around 2Gb large. Needless to say I ran into some memory related issues ;-) At that point I had to react quickly - and these things always happen at night. The solution was to write on-disk data structures and thus avoiding the 2Gb limit. Performance is of course affected by this - but I can safely say that DUDE will unload very large LOB's on a 32bit platform.
3. The one without a system tablespace and just toooo many tables
This one I will also never forget. Losing your system tablespace, and thus your dictionary is a disaster. But no worries - DUDE can identify tables and datatypes using an heuristic algorithm. However, table names, column names, logical column order … that's all stored in the dictionary. So recovery from such a situation involves intimate knowledge of your data, and a lot of guessing to map the flat files to the correct table. I've done it a number of times with a hundred or so tables, always with the developer by my side. It's time consuming and very hard. One time we had a 'lost system tablespace' case. The db in itself was about 68Gb but it was a BAAN ERP system. That means thousands (5000+) of tables !
It took a team of 8 BAAN consultants to rebuild the database in 2 weeks ! I specially changed the code for them so that DUDE would generate BAAN specific flatfiles for easy loading through BAAN's loader tool. This is where the DUDE license really pays off for the customer - one fixed price no matter how long you need it !!!
4. The 'we don't use backups' case… but we're happy with that
One time I got a call from a company who had a SAN crash. They had a 4TB datawarehouse sitting on it, and apparently were unable to restore the database. Because of the size of the DB I suggested to go with the license as clearly, the unload would take some time. A license is fixed priced for as long the unload takes and 4TB… well it can take a while! The next day I get a call back - 'hey this thing really works - we've got another database on that SAN we would like to see unload' - me:'what's the size?' - answer 'about 4TB'… That's not all ! A year later - got a call from the same company different branch. 'We have a db crash - we've used your tool before - we would like to use it again. The DB size is 2TB!". So in 2years, 3 databases and a total of 10TB recovered for the same customer… my favorite customer… ever! They were also the first one who used table compression - so I can safely say my block compression algorithm has been tested on multiple terabytes
As you can see - recovering data can be really stressful, but we take great pride in what we do, and if it means we have to change the code for your specific needs, we'll do it and make it work !
The ORA600 vehicle also made it easier to network and meet new people - which is a bonus as I don't like large crowds. I can definitely say OOW is not for me - but UKOUG is. It's one of the best conferences out there and it allowed me to meet all sorts of interesting new people (you know who you are!). Well, I tasted HOTSOS first which was really great … but I don't like long flights either … and I definitely don't like all male dance parties …. Give me an English pub anytime
Here's a picture of an infamous evening - you'll recognize a certain scottsman in the back, a couple of Danes and a couple of Finns... oh and me !
Anyway - because of the anniversary I'll be giving away the very last ORA600 poloshirts !!!
(to be honest, I have been remodeling the house to accommodate a nursery room as I will become father in December … so everything has got to go!!!
5x size medium (M) , 1x extra large (XL) and 3x extra extra large (XXL)
- the first to respond to firstname.lastname@example.org
will get one !!! (don't forget to mention your size and address!!!)
Update - all polo shirts are gone !!!
Up to the next five years !