JDBC Operations in Spark + Scala

Here is the sample code to create a sample Dataframe from a RDD and then insert that into MySql database. This approved should work for any other relational databases.



Happy Sparking...

Comments

Post a Comment

Popular posts from this blog

Binary Data to Float using Spark SQL

Often when there is massive data communication among the enterprise applications, the data is converted to binary datatype, for reasons like: Faster I/O, Smaller Size, any type of data can be represented, etc. This brings in need to convert binary back to known/readable data type for further analysis/processing etc. For example, here floating point data is represented in Decimal Array[Byte] then stored in Binary format. Ex: [62, -10, 0, 0] Whole floating point equivalent is: 0.480 To achieve above conversion using Apache Spark Sql DataFrames: Voila!
Read more

HBase Table does not exists, but exists

Often we come across this situation with HBase that, table creation says: "ERROR: Table already exists: hbaseFeatures!", however that table wont exist when you list tables. This Often happens when HBase is upgraded to new version, but then there is stale data in Zookeeper. If you are not interested in those tables any more, One way to solve this is to Clear the corresponding data in Zookeeper. Here are some instructions: Now try to create the HBase table, which should succeed. All happy!
Read more

Running Apache Kafka in Windows

Windows Kafka Setup steps: 1) Download Kafka and unzip. 2) Start Zookeeper: kafka_2.10-0.8.2.1\bin\windows\zookeeper-server-start.bat tools\kafka_2.10-0.8.2.1\config\zookeeper.properties 3) Start Kafka Server kafka_2.10-0.8.2.1\bin\windows\kafka-server-start.bat tools\kafka_2.10-0.8.2.1\config\server.properties 4) List topics, to make sure Kafka is up and running kafka_2.10-0.8.2.1\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181 5) Create new Topic, examples: kafka-topics.bat --create --topic sensor1 --replication-factor 1 --zookeeper localhost:2181 --partition 5 6) Produce some sample Kafka Messages: kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.sh --broker-list localhost:9092 --topic sensor1 6) Consumer to Test whether above produced messages are successfully published to Kafka broker: kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic sensor1 --from-beginning 7) Stop Kafka Server: tools\kafka_2....
Read more