The other day I ran across a situation where I was importing an excel file, and the age column I was expecting an age in had the DOB instead. Now I could have handled this on the import side, and probably will in the long run, however there was too much involved and not enough time to get it all working before we needed the data imported. So the search began to find an easy way in excel to convert the DOB to age.
Continue reading
This information is not new, and can be found in the flume cookbook. However these are real world working example showing how to send a file from a flume agent to your collector from the command line. The main use case I have for this, would be to resend a file of events that was rotated while the agent was down, or sending a file that was created before you started monitoring it.
Continue reading
While streaming data to Hadoop with Flume, I have ran into several instances where I get a large amount of 0 byte files.
I am still trying to determine exactly what triggers this to happen, but in the meantime, and for general maintenance and clean up, here is a simple shell command I have come up with to make this happen.
Continue reading
Lately I have been working with and evaluating Flume as a log aggregation engine for our mid-sized web cluster. Flume is relatively simple to get up and running out of the box, and even easier if you take the time to create your own RPM, which I highly recommend if you are going to be deploying to more than a handful of machines.