Saturday, March 15, 2008

Scriptella - Like a Bat Out of ETL!

The Scriptella project has been around a while now. If you haven't checked it out it's really something worth looking at.

Prerequisites:
1. You hate data crap and you always seem to be asked to do it.
2. Brainiac biz guy needs data imported from 3rd party losers: X.
3. You are out of time since you've been browsing articles on
prostitution and Eliot Spitzer all week.
Well fret no more!

Scriptella may be a little lite on the examples, but an hour or so of coding leaves you pretty darn impressed.

You basically setup a few connections in a syntax somewhat related to properties and type only what's necessary to get the job done.

I estimate it takes about 5 minutes of work to have you downloading xml/csv files, parsing them, inserting into the database. Each input is configured in a few seconds and then the beauty of the technology is that it lets you reference variables created from other devices. Available datasources include csv, xml, various jdbc usuals, jexl, janino and a generic 'script' driver.

The script driver is great as it allows the .etl file to use a scripting language to to either invoke Java code or simply create data. By default this is the Javascript Rhino implementation. Badass! Now if I could only figure out how to use it to edit the data on the way in... That would be wicked cool.

Put this all together and this is what I churned out in about 5 minutes of work.

1. Downloaded an xml file
2. Parsed and had them inserted in prepared statements
3. output a simple log on every X records.

Brilliant!

Keep up the good work Scriptella team!

6 comments:

MF Jones said...

Hello Ivan,

Thank you for your review. If you still have an open question on how to modify values in script elements, please post it to our forum. We will try to answer soon.


Best Regards,
Fyodor Kupolov

MF Jones said...

Hello again,

I've written a small transformation example, may be this would help you.

Ivan said...

Great! Thanks for dropping by and putting up the example! It's really amazing Scriptella isn't way more common in development teams. I've seen shops take 3-4 weeks to put together data one-offs in what Scriptella does in about 30 minutes.

MF Jones said...

I agree with you. Before I start writing Scriptella, I've evaluated several ETL/migration tools and found them too bloated from a developer's point of view. They mostly focus on WYSIWYG editors and wizards which is more important for non-programmers.

So Scriptella is definitely an ETL tool for developers who want to control the process, but not to observe it...

Anonymous said...

how scriptella can be used to group the records from input file and send them to separate file.

Anonymous said...

Any way you could post some examples? They seem to be hard to find. I am interested in particular about "logging every X records" like you mentioned. Thanks.