Send Flink's logs to ElasticSearch using Log4j

November 29, 2017

Flink uses slf4j as its logging façade, and log4j as the default logging framework (they support logback too). Logs are accessible via Flink’s UI in the JobManager tab which is good for short-lived jobs but unusable for long-lived, streaming applications. You probably want your logs out of there somewhere else; here’s how you can send them to ElasticSearch so you can access them, say, with Kibana. First, you will need a log4j binding for ElasticSearch; Downfy/log4j-elasticsearch-java-api seems to do the job. ... Read more

Shuffle or pick random lines from a file

November 13, 2017

There comes a day in the life a developer when one needs to choose random lines from a text file. This is useful for a myriad of reasons, like taking a random sample of a CSV file, or shuffling the code of your coworkers, just for fun. There are multiple tools in the UNIX toolbox to solve this problem, but the shuf utility is by far the most elegant: ... Read more

Customizing pyspark app start up script

November 10, 2017

A nice feature of pyspark applications is that you don’t have to use spark-submit manually. Instead, when you instantiate a SparkContext instance, it will take care of running spark-submit ... pyspark-shell for you. Sometimes, however, you may want to customize that launch a bit. For instance, it is useful to tell spark-submit to include specific libraries in the classpath; which is in and of itself a pretty cool feature because you can provide Maven coordinates. ... Read more

JVM notes

November 3, 2017

JVM notes A class file consists of bytecode and a symbol table There are two kinds of types: primitives and references Reference types can be either a dynamically allocated class instance or an array. There is no way to distinguish primitive types within bytecode except for the operands used to manipulate them. Each operand has a different version depending on the type: e.g.: iadd, ladd, fadd and dadd are addition operands for int, long, float and double. ... Read more

Plantasia

September 14, 2017

A piece of art from Mort Garson. A good companion for when you need to go there. Happy trip!

Estimate Gas when using Oraclize

September 12, 2017

Ethereum transactions require you to pay a fee that is measured with gas. Every instruction executed by the EVM costs a certain amount of gas, so before queuing your transaction, you have to specify a gas limit and a gas price. That gas is paid in ether, so for example, if your transaction has a gas limit of 82470 and your gas price is, say, 1 Gwei (0.000000001 ΞTH), then the cost of the transaction would be: ... Read more

God bless Kovan (and Parity)

September 11, 2017

Learning a new technology is always a daunting task, specially when in its early stages. Ethereum is one of such technologies and, soon enough, after starting your journey you get a lot of new terminology/projects thrown at you: Solidity, MetaMask, Mist, Serpent, Rinkeby, Ropsten, Kovan, Parity, Truffle, Geth, etc, etc. As you start wrapping your head around some of those new concepts it is common to focus on specific parts of the whole technology. ... Read more

The Rabbit Hole

September 10, 2017

It seems like no matter how much experience I gain as a developer, and as a human being, time to time I end up going deep into some rabbit hole. This time, in one of those deadly spirals of random clicking on the net I ended up learning about Ethereum. How did I get from there to creating Hugo-based blog hosted on the InterPlanetary File System? I thought I was done with blogging. ... Read more

© 2017 | Powered by Hugo